Re: MiNiFi Java -> NiFi
Matt, this is awesome work and so much of it is invisible to the average eye. Congratulations and thank you for doing this. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 28, 2021, at 16:34, Joe Witt wrote: > > Was thinking the same thing. Id advocate we ditch the minifi jira project. > Just have minifi as a component if nifi jira > > Thanks > >> On Wed, Apr 28, 2021 at 3:45 PM Aldrin Piri wrote: >> >> Awesome work, Matt! >> >> In terms of project management, is there any merit in maintaining the >> MINIFI tracker after the merge is complete versus using components? >> Do you envision subtree releases off the nifi repo for minifi specifically? >> >> On Wed, Apr 28, 2021 at 11:17 AM Steven Matison < >> steven.mati...@datastax.com> >> wrote: >> >>> Way to GO!!! This is awesome stuff. >>> >>> >>> Steven Matison | Data Architect >>> Mobile 321.848.4926 >>> steven.mati...@datastax.com | datastax.com >>> >>>> On Apr 28, 2021, at 2:08 PM, Matt Burgess >> wrote: >>>> >>>> All, >>>> >>>> Now that [1] is merged, MiNiFi Java is now officially in the NiFi >>>> codebase! Thanks to all who participated in discussions, reviews, and >>>> testing. Here are some notes on what this means going forward: >>>> >>>> - All future releases of MiNiFi Java will be in lockstep with NiFi, >>>> the next release version will be 1.14.0 rather than 0.6.0. MiNiFi Java >>>> artifacts will be built and published as part of the NiFi release >>>> process. This will allow MiNiFi to immediately benefit from >>>> improvements/fixes in NiFi components instead of occasionally updating >>>> the NiFi dependency version in MiNiFi Java. >>>> >>>> - Although we will continue to track issues in Jira under the MINIFI >>>> project, PRs should be submitted against the nifi repo's main branch. >>>> If you have an open PR against the nifi-minifi repo, you can port it >>>> to the nifi repo by adding ".patch" to the URL of your PR, downloading >>>> the patch, and applying it to a branch in your fork of the nifi repo, >>>> for example: >>>> >>>> git checkout -b MINIFI-541 >>>> git apply --directory='minifi' ~/195.patch >>>> >>>> Sooner than later we'll make the nifi-minifi repo read-only so no more >>>> PRs can be submitted against it, and we can update NiFi's PR template >>>> to refer to MINIFI Jira cases as well. >>>> >>>> There's still work to be done of course, contributions and PR reviews >>>> are most welcome! >>>> >>>> Regards, >>>> Matt >>>> >>>> [1] https://github.com/apache/nifi/pull/4933 >>> >>> >>
Re: Minor package update to test class
Mark, Thanks for pointing this out. Please do file a Jira for this as it will require a code change. Any commit to the repository needs to go through the Jira/PR/review process, but with small changes like this they can usually get knocked out quickly via coordination between committers. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 28, 2020, at 11:36 AM, Mark Bean wrote: > > A coworker noticed that the package name for TestZooKeeperStateServer.java > is incorrect. Currently, it is > "org.apache.nifi.controller.state.server.zookeeper". However, it should be > "org.apache.nifi.controller.state.server" > > Does this require a full JIRA ticket, PR, etc.? Or is this considered a > documentation update a kind committer could take care of for us? > > Thanks, > Mark
Re: Jira Contributor Access
Hi Parker, I have granted you the contributor role. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 27, 2020, at 5:21 PM, Parker Smith wrote: > > Hello, I'd like to request contributor access to the Nifi project. My Jira > username is divsmith. > > Thanks, > Parker
Re: Parameter Contexts Expression Language scopes
Hi Chad, Parameters were introduced as a way to deprecate (NiFi) variables entirely. I’m not sure that introducing a dependency between the two is a positive step forward. I think there is a separate conversation to be had about allowing parameters access to environment variables, but I think this could introduce problems as parameters are designed for flexibility and portability, and moving from a system where a parameter was actually a pass-through to an environment variable would cause unexpected problems on the destination system. I think the pros and cons of this need to be clearly enumerated and discussed here. Thanks for bringing this up. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 19, 2020, at 9:43 AM, Chad Zobrisky wrote: > > Hello, > > I was configuring an SSL Context Controller Service today and had the > keystores and passwords passed into the container via environment > variables. I thought it would be nice to be able to reference these from > the parameter context. Maybe either giving Parameter Context values the > VARIABLE_REGISTRY scope in the Expression Language, or a new scope for > references external to nifi? > > I think for refreshing the Parameter Context on those external changes, it > would require an edit/re-apply just as it does now, and would have to make > sure it is well documented. > > I'd be interested in creating a PR for this if the idea makes sense and is > acceptable. > > Thanks, > Chad
Re: Dynamic PropertyDescriptors
You could try having custom logic in the #init() method of the processor (or in the method annotated with @OnInitialize for the controller service) which makes an outgoing request, retrieves information from the external source, and then programmatically populates new dynamic properties. The dynamic property mechanism is well-established, and you could theoretically use it to modify the possible values of existing PropertyDescriptor objects as well. There are some buildAllowableValues methods in EncryptContent which do part of this, but obviously they are using data which is static on the JVM and performed at processor class initialization. Just put a placeholder value in initially and delegate the actual population logic to the invoked init() method. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 14, 2020, at 10:47 AM, Jeremy Dyer wrote: > > Bryan, > > That makes sense. Sounds like "dependent properties" isn't quite what I'm > looking for but that is an interesting point about a customValidator. I > will try that. > > As a more concrete example say a ControllerService points to a > unique database table. Depending on the ControllerService/Table selected I > would want to display all of the column names for that Table as valid > options for a PropertyDescriptor. > > What I'm trying to do isn't database related but the example is pretty much > exactly the same idea. > > - Jeremy Dyer > > On Wed, Oct 14, 2020 at 10:52 AM Bryan Bende wrote: > >> Hello, >> >> There is an open PR for "dependent properties" which will hopefully be >> merged soon, but I think it is slightly different than what you are >> asking for. >> >> The idea is that some properties should only be shown based on how >> another property is configured, i.e. changing a strategy property to >> strategy1 should remove any properties relevant to other strategies so >> the user doesn't have to see a bunch of stuff that isn't relevant. >> >> In your scenario it isn't adding/removing whole properties, but >> updating the values of a property. Not sure that has been considered >> yet. >> >> You basically have to implement customValidate and implement logic >> that makes the processor invalid if your two properties have values >> that shouldn't be used together. >> >> -Bryan >> >> On Wed, Oct 14, 2020 at 10:27 AM Jeremy Dyer wrote: >>> >>> Hello, >>> >>> Is it possible to have valid values for a PropertyDescriptor change based >>> on another selection? >>> >>> Ex: I have a ControllerService Property and if I change that >>> ControllerService in the dropdown I would like the valid values for >> another >>> Property to be changed in the UI. >>> >>> I'm thinking this isn't really possible since the logic occurs on the >>> backend after "Apply" has been pressed. I know I can "Apply" and >>> reconfigure to see the new values but would like something more >> immediate. >>> >>> Thanks! >>> Jeremy Dyer >>
Re: CaptureChangeMySQL processor capture maybe wrong when updating more than one table in join statement
Hi Paul,
It appears you may not be subscribed to the mailing list so you did not receive
Matt’s reply one hour after your message. Please subscribe to the list to avoid
this in the future.
You can also read the message here:
https://lists.apache.org/thread.html/re6ad565d566f57a2d76e3707044d08216a1da581206ddf87abe4d2b5%40%3Cdev.nifi.apache.org%3E
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Oct 8, 2020, at 7:46 PM, 杨量（上海-技术部-开发） wrote:
>
> HI, Team
> Could you give me some feedback?
> Thanks
> Paul
> --
> [mailto:liang.y...@feiniu.com]
> : 2020年10月7日 20:29
> : dev@nifi.apache.org
> : Re: CaptureChangeMySQL processor capture maybe wrong when updating more
> than one table in join statement
>
> HI, Team
> I think it should be a bug. Could you plan to fix it or could I register a
> bug for it?
> The wrong issue can be reproduce with 1.9.*, 1.11.4 of NIFI version.
> The mysql version is 5.7.20-log MySQL Community Server (GPL) Wrong behavior
> cause by TABLE_MAP event just got the last one table is the currentTable.
> It should be a map of tableinfo between BEGIN and COMMIT envent.
> When update more table by join statement.
> E.g. the sql statement is:
> UPDATE a a inner join b b m a.id=b.id
> left join c c on a.id=c.id
> set a.name='y'
>,b.name='x'
>,c. name='z';
> The binlog event like belows:
> BEGIN
> TABLE_MAP a
> TABLE_MAP b
> TABLE_MAP c
> Update_rows a
> Update_rows b
> Update_rows c
> COMMIT
> For current codes: The java object currentTable capture the TABLE_MAP c
> event, so, All three update events are resolved to the last table c.
> Can you look at this issue?
>
> BR
> Paul Yang
>
>
> - -
> [mailto:liang.y...@feiniu.com]
> 2020-9-30 9:56
> dev@nifi.apache.org
> CaptureChangeMySQL processor capture maybe wrong when updating more than one
> table in join statement
>
> Hi, Team
>
> I'm working on CaptureChangeMySQL processor with nifi version 1.11.4. I just
> capture update event of one table when updating more than one table in join
> statement.
> The CaptureChangeMySQL behavior is wrong, maybe is a bug. The reason is the
> below code: cannot got the correctly table info when multi Table_map event
> happened.
> The currentTable should be the latest Table_map , so the follow binlog parser
> is going to wrong table name, the table_name is wrong in json.
>
> Is this a known limitation? Could you fix if it is a bug.
>
>case TABLE_MAP:
>// This is sent to inform which table is about to be
> changed by subsequent events
>TableMapEventData data = event.getData();
>
>..
>
>if (!skipTable) {
>TableInfoCacheKey key = new
> TableInfoCacheKey(this.getIdentifier(), data.getDatabase(), data.getTable(),
> data.getTableId());
>if (cacheClient != null) {
>try {
>currentTable = cacheClient.get(key,
> cacheKeySerializer, cacheValueDeserializer);
>
> The belows is my sql.
>
> # UPDATE wdk_sub_orders_drf5089_9 s
> # inner join wdk_main_order_drf5089 m on
> m.main_order_id=s.main_order_id
> # left join wdk_sub_order_mc_drf5089 ms on
> ms.sub_order_id=s.sub_order_id
> # left join wdk_order_promotion_info_drf5089 p on
> p.sub_order_id=s.sub_order_id
> # set
> # s.merchant_id='3'
> # , s.merchant_store_id=1
> # , s.store_sales_type=m.store_sales_type
> # , s.store_channel_id=2
> # , s.presell_type=m.order_type
> # , ms.store_sales_type=m.store_sales_type
> # , ms.order_type=3
> # , p.store_sales_type=4
> # , p.presell_type=m.order_type
> # where
> # s.id between 100382538 and 100382539 and s.source=1 and
> s.merchant_id='2'
> Retrieve All Records: false
> Include DDL Events: false
>
> I got the binlog like belows:
>
>
> #200930 8:46:19 server id 100 end_log_pos 29660 CRC32 0x1e9b0378
> Table_map: `test`.`wdk_sub_orders_drf5089_9` mapped to number 219
> # at 29660
> #200930 8:46:19 server id 100 end_log_pos 29762 CRC32 0xfac7a7b0
> Table_map: `test`.`wdk_sub_order_mc_drf5089` mapped to number 221
> # at 29762
> #200930 8:46:19 server id 100 end_log_pos 29882 CRC32 0x8b437c40
> Table_map: `test`.`wdk_order_promot
Re: request to review NIFI-7738, Provenance Search Events - add reverse query capability
I think the question from Phillip about how to make PR review easier is a great one. For any PR, providing a sample flow which exercises the change is a big time saver already. Demonstrating that unit tests are added/updated, asserting that license considerations have been addressed, and that checkstyle is good are all indicators that the contributor has put comprehensive thought and care into the submission. For interactions with external services (extensions, authentication & authorization, etc.), providing a Docker image with the external services configured or another one-click (or as close to it) way to deploy those dependencies drastically decreases the time and energy a reviewer must expend. I think there is a lot of “hidden cost” that we as committers have not made visible and so for any one PR the contributor is justified in thinking “why aren’t they looking at my code, it’s 20 lines?”. For a committer to review that though, requires pausing any active development work they are doing, checking out a new branch, building the whole project (~45 minutes with tests & checkstyle on decent hardware), deploying and configuring any external dependencies, running the application and exercising the change, checking for edge cases, etc. All of that is independent from the actual code review process for logic, vulnerabilities, performance, adherence to best practices & design principles, edge cases, and supportability & maintainability by the community at large. There are a number of times where the preferable path is for a contributor to make the code available publicly in their own GitHub repository. This sidesteps the community review aspect, but also means that they are not reliant on the community at large for release and development timelines, nor subject to a response of “you are the only person who understands how [external component X] works, so we as a community cannot take on the support and maintenance responsibilities of this contribution." Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 8, 2020, at 9:52 AM, Nissim Shiman wrote: > > Hello Joe, Phillip and Pierre, > I am really happy we are having this discussion. > I (and very likely other contributers) was under the impression that only > reviews by committers/PMC members (i.e. more senior nifi dev community > members) had any real weight, so that is why I have not been reviewing > standing PRs. > > > So this discussion is bringing out out that anyone can review/test open PRs > and once any issues in the PRs have been smoothed out that it shouldn't be > much of an issue to have it committed at that point. > > Actually, it looks like a fellow contributer, Mark Bean, has already noticed > this conversation and has already reviewed my PR and has made some useful > comments. So we are already seeing results. (and thank you Mark!) > > I plan to start reviewing/testing PRs. > Thank You, > Nissim Shiman >On Thursday, October 8, 2020, 12:17:13 PM EDT, Joe Witt > wrote: > > Hello > > Anyone can do a review of the code and build and feasibility of a change. > It only requires commit privileges to actually merge something. > > Doing the hard work of reviewing/testing the PRs is a *great* way to earn > merit. > > Much of the PRs that tend to hang around are from people contributing a > specific thing and those are some of the most challenging. We as a > community then take on support of those things for function, License and > Notice adherence, etc.. On the other hand, these folks are also the ones > that can often start contributing elsewhere and building merit and earning > commit and even PMC status. So we have to invest one way or another. But > for sure this is not easy to do well as time has shown. > > Nissim, in your case the contribution is probably cool/good/useful. What > could help you get more attention on it is helping with bug fixes or > reviewing other peoples contributions/etc.. Not saying anyone is checking > whether you do that or not but if they see more of your effort it *does* > help improve likelihood of getting merge attention. Not required...just > sharing another perspective. > > Regardless thanks for the contribution and hopefully someone is able to > review soon. > > Thanks > > On Thu, Oct 8, 2020 at 9:00 AM Phillip Grenier wrote: > >> Pierre, >> >> I think this discussion brings up a valid conversation point. At some point >> a PMC member needs to approve the merge request, so from a contributors >> level what can we do to make that merge both easier and/or more likely to >> happen. That and how the community can help filter down t
Re: NIFI version 1.9.2-RC2 (UNCLASSIFIED)
Hi Robert, The “-RC2” part of the version number indicates the release candidate of the version. In this case, RC 2 is the candidate which was released, so your version is effectively 1.9.2. The documentation hosted on our website changes for the main guides with each version, but previous versions of specific component documentation is maintained for some time (e.g. UpdateAttribute 1.9.2 [1]). However, all of the documentation for the specific version you are using is always bundled with the application. To access it, from the main canvas view use the global menu in the top right corner and select “Help” for access to all the guides and component documentation, or right click on any processor on the canvas and select “Usage” to see that specific documentation. I am curious as to how & why you are using a release candidate of the software. Did you compile the binaries yourself from source code you checked out, or did you retrieve the “released” source code and convenience binaries from our downloads page [2]? The version you are referencing is over a year old, so I would encourage you to upgrade and use a formally released version if possible. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.9.2/org.apache.nifi.processors.attributes.UpdateAttribute/index.html [2] https://nifi.apache.org/download.html Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 7, 2020, at 10:02 AM, Moore, Robert W CTR (USA) > wrote: > > CLASSIFICATION: UNCLASSIFIED > > Good Afternoon, > > My version of Apache NIFI is version 1.9.2-RC2 > > I cannot find a user/admin document for that version. Is it possible for you > to send me a PDF copy? I am new to using NIFI and I am getting configuration > errors that I cannot fix. Thanks in advance > > If you are not the POC for this request, please forward to someone who can > help me… > > > > Regards, > Robert Moore > System Administrator Sr / DCGS-FSE > 470th MI BDE, I2TC > robert.w.moore20@mail.mil <mailto:robert.w.moore20@mail.mil> > robert.w.moore20@mail.smil.mil <mailto:robert.w.moore20@mail.smil.mil> > 210-295-1139 work (ask for Robert) > > “Please be advised I am a contractor for INSCOM G6/GISA. My correspondence is > for > coordination and information gathering purposes only and in no way binds or > obligates the Government contractually.” > > > > CLASSIFICATION: UNCLASSIFIED
[ANNOUNCE] Apache NiFi CVE-2020-9486, CVE-2020-9487, CVE-2020-9491, CVE-2020-13940
Apache NiFi PMC would like to announce the discovery and resolution of CVE-2020-9486, CVE-2020-9487, CVE-2020-9491, and CVE-2020-13940. These issues have been resolved and a new version of the Apache NiFi project was released in accordance with the Apache Release Process. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Fixed in Apache NiFi 1.12.0 (Released: August 18, 2020) CVE-2020-9486: Apache NiFi information disclosure in logs Severity: Important Versions Affected: Apache NiFi 1.10.0 - 1.11.4 Description: The NiFi stateless execution engine produced log output which included sensitive property values. When a flow was triggered, the flow definition configuration JSON was printed, potentially containing sensitive values in plaintext. Mitigation: Implemented Argon2 secure hashing to provide a deterministic loggable value which does not reveal the sensitive value. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Andy LoPresto and Pierre Villard. CVE-2020-9487: Apache NiFi denial of service Severity: Important Versions Affected: Apache NiFi 1.0.0 - 1.11.4 Description: The NiFi download token (one-time password) mechanism used a fixed cache size and did not authenticate a request to create a download token, only when attempting to use the token to access the content. An unauthenticated user could repeatedly request download tokens, preventing legitimate users from requesting download tokens. Mitigation: Disabled anonymous authentication, implemented a multi-indexed cache, and limited token creation requests to one concurrent request per user. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Dennis Detering (IT Security Consultant at Spike Reply). CVE-2020-9491: Apache NiFi use of weak TLS protocols Severity: Critical Versions Affected: Apache NiFi 1.2.0 - 1.11.4 Description: The NiFi UI and API were protected by mandating TLS v1.2, as well as listening connections established by processors like ListenHTTP, HandleHttpRequest, etc. However intracluster communication such as cluster request replication, Site-to-Site, and load balanced queues continued to support TLS v1.0 or v1.1. Mitigation: Refactored disparate internal SSL and TLS code, reducing exposure for extension and framework developers to low-level primitives. Added support for TLS v1.3 on supporting JVMs. Restricted all incoming TLS communications to TLS v1.2+. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Juan Carlos Sequeiros and Andy LoPresto. CVE-2020-13940: Apache NiFi information disclosure by XXE Severity: Low Versions Affected: Apache NiFi 1.0.0 - 1.11.4 Description: The notification service manager and various policy authorizer and user group provider objects allowed trusted administrators to inadvertently configure a potentially malicious XML file. The XML file has the ability to make external calls to services (via XXE). Mitigation: An XML validator was introduced to prevent malicious code from being parsed and executed. Users running any previous NiFi release should upgrade to the latest release. Credit: This issue was discovered by Matt Burgess and Andy LoPresto. For more information: https://nifi.apache.org/security.html <https://nifi.apache.org/security.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
Re: Jira contributor access
Hi Kristen, I added you to the contributor role. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 29, 2020, at 11:23 AM, Kristen Guthier > wrote: > > Requesting Jira contributor access for username kkguthier
Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX OS
Sorry, I had accidentally replied directly to the user and not the list with my previous email. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 28, 2020, at 9:24 AM, Andy LoPresto wrote: > > Running the toolkit in standalone mode on each independent machine won’t work > because each generated certificate will be signed by a different certificate > authority. If using standalone mode, all of the commands must be run in the > same directory on a single machine and the certificates copied out to each > node. If you must run the command on each separate machine, use the > client/server mode. The walkthrough makes clear that this particular example > is using multiple NiFi instances on the same physical machine. > > The log messages you posted are not errors and are expected. The settings in > nifi.properties around cluster election determine how long the cluster will > wait before deciding the election is complete. If you only have two nodes, > you can set the max candidates value to 2, and the cluster won’t wait for > additional nodes after 2 have joined. > > There is no SLA or response time expectation for these mailing lists. They > are staffed by members of the open source community donating their time and > expertise of their own will. If you have critical delivery issues, you should > investigate a vendor for paid support. > > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Sep 27, 2020, at 3:54 PM, Abiodun Adegbile > <mailto:aadegb...@bluechiptech.biz>> wrote: >> >> Hello Andy, >> >> I used this link below >> >> https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-provided-certificates >> >> <https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-provided-certificates> >> Apache NiFi Walkthroughs >> <https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html#securing-nifi-with-provided-certificates> >> Apache NiFi is an easy to use, powerful, and reliable system to process and >> distribute data. It supports powerful and scalable directed graphs of data >> routing, transformation, and system mediation logic. >> nifi.apache.org <http://nifi.apache.org/> >> A quick question, when generating the certificate for the server >> (./bin/tls-toolkit.sh standalone -n 'node1.nifi' -c 'ca.nifi'), do i >> generate on each separate server like this (./bin/tls-toolkit.sh standalone >> -n 'node[2-3].nifi') or will this two be generated from node1 toolkits.sh >> for all the other nodes since they are not on the same host but different >> host servers? >> >> By the way, this was the error i got after trying to setup a 2node nifi >> cluster on just 2 separate servers: >> >> >> I really look forward to your response ASAP within the hour as this is >> critically urgent for my project delivery. >> >> Thanks >> >> >> >> KEEP SAFE, STAY SAFE & STAY STRONG >> >> I celebrate you... >> >> ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP) >> Database Administrator/Infrastructure Services >> Bluechip Technologies Limited >> Plot 9B, Onikoyi Lane >> Ikoyi. Lagos State. >> Mobile : +234 806 206 9970 >> Website : www.bluechiptech.biz <http://www.bluechiptech.biz/> >> >> >> ...2016 Winner Oracle Transformational Deal of the Year - Nigeria. >> >> >> From: Andy LoPresto mailto:alopre...@apache.org>> >> Sent: Thursday, September 24, 2020 4:05 PM >> To: Abiodun Adegbile > <mailto:aadegb...@bluechiptech.biz>> >> Cc: dev@nifi.apache.org <mailto:dev@nifi.apache.org> > <mailto:dev@nifi.apache.org>> >> Subject: Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON >> LINUX OS >> >> I’m not sure I understand your request then. Which walkthroughs have you >> tried and how have they failed? Are you getting specific exceptions? >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> He/Him >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Sep 24, 2020, at 2:03 AM, Abiodun Adegbile >> <mailto:aadegb...@bluechiptech.biz>> wrote: >>> >>> Yes Andy, i have consulted Googl
Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX OS
I’m not sure I understand your request then. Which walkthroughs have you tried and how have they failed? Are you getting specific exceptions? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 24, 2020, at 2:03 AM, Abiodun Adegbile > wrote: > > Yes Andy, i have consulted Google for resource on the problem area and have > tried a couple of workarounds. > > > > KEEP SAFE, STAY SAFE & STAY STRONG > > I celebrate you... > > ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP) > Database Administrator/Infrastructure Services > Bluechip Technologies Limited > Plot 9B, Onikoyi Lane > Ikoyi. Lagos State. > Mobile : +234 806 206 9970 > Website : www.bluechiptech.biz <http://www.bluechiptech.biz/> > > > ...2016 Winner Oracle Transformational Deal of the Year - Nigeria. > > From: Andy LoPresto > Sent: Thursday, September 24, 2020 12:06 AM > To: dev@nifi.apache.org ; Abiodun Adegbile > > Subject: Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX > OS > > Hi Adegbile, > > If you are not registered for the mailing list, you will not receive the > responses. Community members did reply to your initial email but I suspect > you were not aware. > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Sep 23, 2020, at 1:22 PM, Russell Bateman > <mailto:r...@windofkeltia.com>> wrote: >> >> You have treated Google as your friend and worked through the dozen or so >> examples by other folk doing this, right? >> >> --just a suggestion. >> >> On 9/23/20 1:21 PM, Abiodun Adegbile wrote: >>> */Hello Team,/* >>> >>> Still looking forward to your reply. >>> / >>> / >>> >>> /I got this though after i tried setting up the 2-node cluster on two >>> separate physical servers vms/ >>> >>> */ >>> >>> /* >>> >>> */ >>> /* >>> >>> Your prompt response will be highly appreciated. >>> >>> */ >>> /* >>> >>> */KEEP SAFE, STAY SAFE & STAY STRONG/* >>> >>> / >>> / >>> >>> /I celebrate you.../ >>> >>> >>> /*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/ >>> >>> */Database Administrator/Infrastructure Services/* >>> >>> */Bluechip Technologies Limited/* >>> >>> */Plot 9B, Onikoyi Lane/* >>> >>> */Ikoyi. Lagos State./* >>> >>> */Mobile : +234 806 206 9970/* >>> >>> */Website : www.bluechiptech.biz <http://www.bluechiptech.biz/> >>> <http://www.bluechiptech.biz <http://www.bluechiptech.biz/>>/* >>> >>> */ >>> /* >>> >>> .../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./ >>> >>> >>> >>> *From:* Abiodun Adegbile >>> *Sent:* Sunday, September 20, 2020 2:33 AM >>> *To:* dev@nifi.apache.org <mailto:dev@nifi.apache.org> >> <mailto:dev@nifi.apache.org>> >>> *Subject:* APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX >>> OS >>> */Hello Team,/* >>> >>> Good to write to you. >>> >>> I am planning to implement a NIFI cluster installation for two separate >>> physical linux servers(to-be-clustered) for my test lab. I was hoping you >>> could send a walkthrough documentation for this. >>> >>> Many thanks for your help. Will appreciate your prompt response. >>> >>> */ >>> /* >>> >>> */ >>> /* >>> >>> */KEEP SAFE, STAY SAFE & STAY STRONG/* >>> >>> / >>> / >>> >>> /I celebrate you.../ >>> >>> >>> /*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/ >>> >>> */Database Administrator/Infrastructure Services/* >>> >>> */Bluechip Technologies Limited/* >>> >>> */Plot 9B, Onikoyi Lane/* >>> >>> */Ikoyi. Lagos State./* >>> >>> */Mobile : +234 806 206 9970/* >>> >>> */Website : www.bluechiptech.biz <http://www.bluechiptech.biz/> >>> <http://www.bluechiptech.biz <http://www.bluechiptech.biz/>>/* >>> >>> */ >>> /* >>> >>> .../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./
Re: [VOTE] Release Apache NiFi 1.12.1 (rc2)
+1, binding. Verified the checksums, the GPG signature, the LICENSE/NOTICE/README.md files, the full build, and started a simple node on Mac OS X 10.15.6. Specifically verified the results of NIFI-7804 [1] and examined the resulting nifi-security-utils and nifi-security-utils-api JAR files. ..n/nifi-1.12.1 main ● ls -alGFh work/nar/extensions/nifi-standard-nar-1.12.1.nar-unpacked/NAR-INF/bundled-dependencies/ | grep nifi-security -rw-r--r--1 alopresto staff 180K Sep 23 16:56 nifi-security-utils-1.12.1.jar ..n/nifi-1.12.1 main ● ls -alGFh work/nar/extensions/nifi-standard-services-api-nar-1.12.1.nar-unpacked/NAR-INF/bundled-dependencies/ | grep nifi-security -rw-r--r-- 1 alopresto staff13K Sep 23 16:56 nifi-security-utils-api-1.12.1.jar Thanks for taking this effort on, Joe. [1] https://issues.apache.org/jira/browse/NIFI-7804 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 23, 2020, at 3:10 PM, Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.12.1. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1170 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.1/ > > A helpful reminder on how the release candidate verification process works: > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.12.1-RC2 > The Git commit ID is accfaa3034fa5c3ef55d6402ac31e500247100f9 > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=accfaa3034fa5c3ef55d6402ac31e500247100f9 > > Checksums of nifi-1.12.1-source-release.zip: > SHA256: 7735c632c6795bb0d299631454bd81006db33d51192cacc1404a5c9779607802 > SHA512: > e7f06afc7617df7e3325bce8e34d1d78c4d130c40135661e59ae0eabef50df888759a125dae3ab9556fc940d621c695a50e674ebad8c3066716bfb5fbd27c1c4 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 39 issues were closed/resolved for this release: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348757 > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.1 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.12.1 > [ ] +0 no opinion > [ ] -1 Do not release this package because...
Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX OS
Hi Adegbile, If you are not registered for the mailing list, you will not receive the responses. Community members did reply to your initial email but I suspect you were not aware. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 23, 2020, at 1:22 PM, Russell Bateman wrote: > > You have treated Google as your friend and worked through the dozen or so > examples by other folk doing this, right? > > --just a suggestion. > > On 9/23/20 1:21 PM, Abiodun Adegbile wrote: >> */Hello Team,/* >> >> Still looking forward to your reply. >> / >> / >> >> /I got this though after i tried setting up the 2-node cluster on two >> separate physical servers vms/ >> >> */ >> >> /* >> >> */ >> /* >> >> Your prompt response will be highly appreciated. >> >> */ >> /* >> >> */KEEP SAFE, STAY SAFE & STAY STRONG/* >> >> / >> / >> >> /I celebrate you.../ >> >> >> /*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/ >> >> */Database Administrator/Infrastructure Services/* >> >> */Bluechip Technologies Limited/* >> >> */Plot 9B, Onikoyi Lane/* >> >> */Ikoyi. Lagos State./* >> >> */Mobile : +234 806 206 9970/* >> >> */Website : www.bluechiptech.biz <http://www.bluechiptech.biz>/* >> >> */ >> /* >> >> .../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./ >> >> >> >> *From:* Abiodun Adegbile >> *Sent:* Sunday, September 20, 2020 2:33 AM >> *To:* dev@nifi.apache.org >> *Subject:* APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX >> OS >> */Hello Team,/* >> >> Good to write to you. >> >> I am planning to implement a NIFI cluster installation for two separate >> physical linux servers(to-be-clustered) for my test lab. I was hoping you >> could send a walkthrough documentation for this. >> >> Many thanks for your help. Will appreciate your prompt response. >> >> */ >> /* >> >> */ >> /* >> >> */KEEP SAFE, STAY SAFE & STAY STRONG/* >> >> / >> / >> >> /I celebrate you.../ >> >> >> /*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/ >> >> */Database Administrator/Infrastructure Services/* >> >> */Bluechip Technologies Limited/* >> >> */Plot 9B, Onikoyi Lane/* >> >> */Ikoyi. Lagos State./* >> >> */Mobile : +234 806 206 9970/* >> >> */Website : www.bluechiptech.biz <http://www.bluechiptech.biz>/* >> >> */ >> /* >> >> .../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./ >> >> >
Re: Requesting Jira Contributor Access
Hi David, I’ve added you to the contributors role. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 22, 2020, at 3:10 PM, Exception Factory > wrote: > > Hello, > > I would like contributor access to the NiFi Jira project. My Jira username > is exceptionfactory. > > Regards, > David Handermann
Re: [VOTE] Release Apache NiFi 1.12.1
+1 (binding) Ran through normal release verification process and specifically checked the resolutions to NIFI-7730 [1] and NIFI-7767 [2]. [1] https://issues.apache.org/jira/browse/NIFI-7730 [2] https://issues.apache.org/jira/browse/NIFI-7767 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Sep 8, 2020, at 2:20 PM, Matt Burgess wrote: > > +1 (binding) Release this package as nifi-1.12.1 > > Ran through the release helper and various flows. Thanks for RM'ing Joe! > > On Tue, Sep 8, 2020 at 1:00 PM Joe Witt wrote: >> >> Hello, >> >> I am pleased to be calling this vote for the source release of Apache NiFi >> 1.12.1. >> >> The source zip, including signatures, digests, etc. can be found at: >> https://repository.apache.org/content/repositories/orgapachenifi-1167 >> >> The source being voted upon and the convenience binaries can be found at: >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.1/ >> >> A helpful reminder on how the release candidate verification process works: >> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate >> >> The Git tag is nifi-1.12.1-RC1 >> The Git commit ID is e8f708b3945183b7ab1b356c00d0fcbd929d6163 >> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=e8f708b3945183b7ab1b356c00d0fcbd929d6163 >> >> Checksums of nifi-1.12.1-source-release.zip: >> SHA256: 6fa7e4389ff33957133292e9749876affe1258e6f30d4e60b53e8c2d01d032cb >> SHA512: >> fee615ee885b05a290a4d6c7399ecf0b70a93cc1d5d9b019c7840ceb89d3279a540da4de75765c589c301fff751e62c9e15cc33d078b5fad03bbbd28e74f8ea9 >> >> Release artifacts are signed with the following key: >> https://people.apache.org/keys/committer/joewitt.asc >> >> KEYS file available here: >> https://dist.apache.org/repos/dist/release/nifi/KEYS >> >> 19 issues were closed/resolved for this release: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348757 >> >> Release note highlights can be found here: >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.1 >> >> The vote will be open for 72 hours. >> Please download the release candidate and evaluate the necessary items >> including checking hashes, signatures, build >> from source, and test. Then please vote: >> >> [ ] +1 Release this package as nifi-1.12.1 >> [ ] +0 no opinion >> [ ] -1 Do not release this package because...
Re: From one flowfile to two...
Russell,
Glad you found a working solution. Maybe it would be better for you to write up
your findings and share them with a broader audience. I have often seen the
best explanations are written by people who were recently in the “how do I do
X?” state, as they are closest to the problem and can walk through their
process of gathering understanding. Someone who works on these methods day in
and day out may not write for the appropriate audience or explain the
experience as well.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Aug 27, 2020, at 10:10 AM, Russell Bateman wrote:
>
> I needed to get back here...
>
> I took this advice to heart and finished my processor. Thanks to Matt and
> Mark for all their suggestions! They cleared up a few things. There was one
> bug in the code that was mine, small, but significant in its effect on the
> rest. That mistake also explained why I thought the uuidwas identical between
> at least two of the cloned flowfiles. What I would wish for, and am probably
> not strong enough to write, would be a synthesis of the session methods
> read() and write() and how best to use them (one-to-one, one-to-many, etc.).
> Javadoc is too paratactic by nature, the NiFi Developer's Guide almost silent
> on these methods. If it were not for the many existing examples using these
> methods, it would be hard to learn to do even simple things. I did look for
> something closer to what I needed to do, but unsuccessfully.
>
> Thanks again. If anything, the NiFi mailing lists are a place both for great
> information and being treated well.
>
> Russ
>
> On 8/25/20 12:24 PM, Mark Payne wrote:
>> Russ,
>>
>> Several comments here. I’ve included them inline, below.
>>
>> Hope it’s helpful.
>>
>> Thanks
>> -Mark
>>
>>
>>> On Aug 25, 2020, at 2:09 PM, Russell Bateman wrote:
>>>
>>> Thanks for your suggestions, Matt.
>>>
>>> I decided to keep the original flowfile only upon failure. So, I have the
>>> embedded-document file and the serialized POJOs created from processing the
>>> non embedded-document part as the result if successful. (Condensed code at
>>> end...)
>>>
>>> Now I have three questions...
>>>
>>> 1. I seem not to have placated NiFi with the assurance that I have
>>> transferred or disposed of all three flowfiles suitably. I get:
>>>
>>> java.lang.AssertionError:
>>> org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot
>>> commit session because the following FlowFiles have not been removed or
>>> transferred: [2]
>>>> This is probably because at the end of the block, you catch Exception and
>>>> then route the original FlowFile to failure. But you’ve already cloned it
>>>> and didn’t deal with the clone.
>>> *Which of the three flowfiles does [2] refer to? Or does it just mean I
>>> botched two flowfiles. *
>>>
>>> 2. session.clone()generates a new flowfile with the identical uuid. I don't
>>> think I want the result to be two flowfiles with the same uuid. I am
>>> binding them together so I can associate them later using attribute
>>> embedded-document. *Should I/How do I force cloning to acquire new
>>> **uuid**s?*
>>>>> This appears to actually be a bug in the mock framework. It *should* have
>>>>> a unique uuid, and would in a running NiFi instance. Feel free to file a
>>>>> Jira for that.
>>> 3. A question on theory... *Wouldn't all of this cloning be expensive* and
>>> I should just clone for one of the new files and then mangle the original
>>> flowfile to become the other?
>>>> session.clone() is not particularly expensive. It’s just creating a new
>>>> FlowFile object. It doesn’t clone the FlowFile’s contents.
>> That said, it is probably more appropriate to call session.create(flowFile),
>> rather than session.clone(flowFile). It makes little difference in practice
>> but what you’re really doing is forking a child, and that will come across
>> more cleanly in the Provenance lineage that is generated if using
>> session.create(flowFile).
>>
>> Additional comments in code below.
>>
>>
>>> Thanks,
>>> Russ
>>>
>>>
>>> @Override
>>> public void onTrigger( final ProcessContext context, final ProcessSession
>>> session ) throws ProcessException
>>> {
>>> FlowFile flowfile = se
Re: "Jira contributor access" to username : ashinde
Hi Akash, I added you to the contributors role. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 27, 2020, at 8:58 AM, Akash Shinde wrote: > > Team, > > Please allow Jira contributor access to my apache jira username: ashinde > > Thank you in advance. > > -Akash
Re: Nifi Authentication
Ashar, You are not subscribed to the mailing list so you are not seeing replies. I replied to you last Thursday stating that there is no support for basic authentication for Apache NiFi. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 17, 2020, at 14:04, Ashar Azmat wrote: > > Hello team, > > I just wanted to follow up on my previous request. > > Regards, > Ashar > > From: Ashar Azmat > Sent: Thursday, August 13, 2020 2:57 PM > To: dev@nifi.apache.org > Cc: Wesley Philip > Subject: Nifi Authentication > > Hello team, > > Hope this email finds you well. We had a question in regards to > authentication in relation to the Nifi User Interface. > We would like to enable authentication by using basic authentication. Is > there any native basic authentication support or do we have to use an > alternative authentication mechanism such as Kerberos or LDAP? > > We would appreciate your help in this regard. > > Best, > Ashar > ::DISCLAIMER:: > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. E-mail transmission is not > guaranteed to be secure or error-free as information could be intercepted, > corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses > in transmission. The e mail and its contents (with or without referred > errors) shall therefore not attach any liability on the originator or HCL or > its affiliates. Views or opinions, if any, presented in this email are solely > those of the author and may not necessarily reflect the views or opinions of > HCL or its affiliates. Any form of reproduction, dissemination, copying, > disclosure, modification, distribution and / or publication of this message > without the prior written consent of authorized representative of HCL is > strictly prohibited. If you have received this email in error please delete > it and notify the sender immediately. Before opening any email and/or > attachments, please check them for viruses and other defects. >
Re: [VOTE] Release Apache NiFi 1.12.0
+1 (binding). Verified the release info (signature, digests [equal to provided digests; different from email as noted in other thread], etc.), the full build, and some flows exercising new cryptographic metadata. Great work on this enormous release everyone. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 13, 2020, at 1:31 PM, Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.12.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1165 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.0/ > > A helpful reminder on how the release candidate verification process works: > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.12.0-RC1 > The Git commit ID is 4d940bb151eb8d250b0319318b96d23c4a9819ae > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=4d940bb151eb8d250b0319318b96d23c4a9819ae > > Checksums of nifi-1.12.0-source-release.zip: > SHA256: a67ecb34f32cc1f070ebb006b6f05456a2ac663b12f708eadac67754194a6c63 > SHA512: > 2e04235c4d49a76319af7756289ce11554a412bf5f7dcb6dc3915fc931df9f067142820c297e83bc36cb1079fb8384794ef457a20dd00568761eed6621701953 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 335 issues were closed/resolved for this release: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12346778 > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.12.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because...
Re: Nifi Authentication
There is no support for Basic Authentication to access the NiFi UI / API. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 13, 2020, at 11:57 AM, Ashar Azmat wrote: > > Hello team, > > Hope this email finds you well. We had a question in regards to > authentication in relation to the Nifi User Interface. > We would like to enable authentication by using basic authentication. Is > there any native basic authentication support or do we have to use an > alternative authentication mechanism such as Kerberos or LDAP? > > We would appreciate your help in this regard. > > Best, > Ashar > ::DISCLAIMER:: > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. E-mail transmission is not > guaranteed to be secure or error-free as information could be intercepted, > corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses > in transmission. The e mail and its contents (with or without referred > errors) shall therefore not attach any liability on the originator or HCL or > its affiliates. Views or opinions, if any, presented in this email are solely > those of the author and may not necessarily reflect the views or opinions of > HCL or its affiliates. Any form of reproduction, dissemination, copying, > disclosure, modification, distribution and / or publication of this message > without the prior written consent of authorized representative of HCL is > strictly prohibited. If you have received this email in error please delete > it and notify the sender immediately. Before opening any email and/or > attachments, please check them for viruses and other defects. >
Re: Live development of custom processors and JAVA_HOME--confirm best practice
Russell, To develop and test against multiple versions of Java, I (and I’m aware other committers) use jenv [1] which is a version manager for Java. You can set a specific version for global/shell/local environments, and it allows switching very quickly. I generally run these two commands to switch: $ jenv shell 1.8.0 # switch to JDK 8 $ jjh # alias defined below # In my .zshrc (or whatever shell configuration file you use) alias jjh='jenv version && jenv macos-javahome && jenv_set_java_home && jenv doctor’ alias jenv_set_java_home='export JAVA_HOME="$HOME/.jenv/versions/`jenv version-name`" && echo "JAVA_HOME=$JAVA_HOME”' Setting the $JAVA_HOME ensures that mvn picks up the right Java version when it runs. [1] https://www.jenv.be/ <https://www.jenv.be/> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 13, 2020, at 10:48 AM, Russell Bateman wrote: > > When installing NiFi in production, Ansible can be used to set up JAVA_HOME. > There is zero problem for users of NiFi. > > However, from a development host, given the now rapid cadence of Java > releases, we sometimes run into problems launching a private installation of > NiFi in the course of testing or debugging our custom processors because > tools used minute-to-minute and all day long like IntelliJ IDEA (and others) > march on requiring later and later Java versions. > > I found an old JIRA issue that suggests a solution for NiFi 0.1.0, that of > working around the problem by setting java= in /conf/bootstrap.conf/ to > point to a valid Java 1.8 JRE/JDK. This sounds good to me, but the version is > very old. > > Is this still best practice? >
Re: [ANNOUNCE] New Apache NiFi Committer Marton Szasz
Congratulations Marton. Thanks for all the great contributions so far and looking forward to many more. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 3, 2020, at 7:37 AM, Joe Witt wrote: > > Congrats - your efforts around minifi and nifi at large are > greatly appreciated > > On Mon, Aug 3, 2020 at 7:35 AM Tony Kurc wrote: > >> Congrats Marton! >> >> On Mon, Aug 3, 2020 at 4:02 AM Arpad Boda wrote: >> >>> Apache NiFi community, >>> >>> On behalf of the Apache NiFI PMC, I am very pleased to announce that >> Marton >>> has accepted the PMC's invitation to become a committer on the Apache >> NiFi >>> project. We greatly appreciate all of Marton's hard work and generous >>> contributions to the project. We look forward to continued involvement in >>> the project. >>> >>> Marton had more than 100 contributions to MiNiFi C++ this year in various >>> areas: from Windows-specific memory leaks to nice new features and a lot >> of >>> code reviews. He also showed active presence on the mailing list helping >>> out the community. >>> >>> Welcome and congratulations! >>> >>
Re: Failing to update custom processor properties names, displayNames, etc.
Russell,
Have you verified this is not a browser caching issue? They are pernicious and
it sounds like this could be an example. If you’re sure it’s not, verify the
API calls using your browser’s Developer Tools to see what properties are
actually being returned by the server when inspecting a component to see if the
correct values are present. If so, it’s likely a UI bug, and if not, the new
code is not being properly loaded/used by NiFi’s server application.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 17, 2020, at 3:48 PM, Russell Bateman wrote:
>
> If I have changed a custom processor's PropertyDescriptor.nameand/or
> .displayName,including changes I have made to my /additionalDetails.html,
> /and I have:
>
> - removed that processor from my test flow or removed flow.xml.gz altogether
> - removed my NAR from /${NIFI_ROOT}///custom-lib/ and bounced NiFi with the
> new custom NAR copied
>
> yet nothing changes (in usage or in additional details), what am I
> overlooking?
>
> Let's set aside the vaguely confusing semantic distinction between nameand
> displayName, I just want NiFi to forget my new custom processor completely
> and then accept my new version as if brand new including all the changes I
> have made.
>
> Thanks for any suggestions.
Re: [VOTE] Release Apache NiFi Registry 0.7.0
+1 binding * Verified the artifact signatures and digests (I don’t think the MD5 and SHA1 digests need to be published anymore). * Verified the build process and contrib check. * Verified the LICENSE and NOTICE files. * Deployed a NiFi Registry 0.7.0 instance and connected to it with NiFi 1.12.0-SNAPSHOT using current main branch. * Deployed a secure NiFi Registry 0.7.0 instance and connected to it with a secure NiFi 1.12.0-SNAPSHOT instance. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 16, 2020, at 10:11 AM, Chad Zobrisky wrote: > > +1 non-binding. > > Verified full clean build with contrib check on Java 8, commit tag and > hash. diffed sources and looked good. > License + Notice looked good. > Ran local docker image and created a few buckets on a non-secure > nifi/registry. > > All looks good. > > > On Wed, Jul 15, 2020 at 3:36 PM Joe Witt wrote: > >> +1 binding. Verified full clean build w contrib check. Sources match >> commit tag. Hashes/Signature good. L looks good. Builds/runs on Java 11. >> Version control to/from NiFi on non secured config. >> >> On Wed, Jul 15, 2020 at 8:32 AM Bryan Bende wrote: >> >>> Hello, >>> >>> I am pleased to be calling this vote for the source release of Apache >> NiFi >>> Registry 0.7.0. >>> >>> The source zip, including signatures, digests, etc. can be found at: >>> https://repository.apache.org/content/repositories/orgapachenifi-1161 >>> >>> The source being voted upon and the convenience binaries can be found at: >>> >>> >> https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.7.0/ >>> >>> A helpful reminder on how the release candidate verification process >> works: >>> >>> >> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate >>> >>> The Git tag is nifi-registry-0.7.0-RC1 >>> The Git commit ID is c8f26039712354b94c4d458b7ea491316c6facac >>> >>> >> https://gitbox.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=c8f26039712354b94c4d458b7ea491316c6facac >>> >>> Checksums of nifi-registry-0.7.0-source-release.zip: >>> SHA256: acc6b21444d229d78b8a604750231e1c7e80495c9cccfb59ec043ebe952fd2a8 >>> SHA512: >>> >>> >> 428839e0c861095547e328fcfda42cb3f2a87c0b03f7205d6878f8b7267975fe2ad051d4b5f980cac8f47a249c8ee21686c8d909f275454e25d8a654099bd683 >>> >>> Release artifacts are signed with the following key: >>> https://people.apache.org/keys/committer/bbende.asc >>> >>> KEYS file available here: >>> https://dist.apache.org/repos/dist/release/nifi/KEYS >>> >>> 19 issues were closed/resolved for this release: >>> >>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320920=12346077 >>> >>> Release note highlights can be found here: >>> >>> >> https://cwiki.apache.org/confluence/display/NIFIREG/Release+Notes#ReleaseNotes-NiFiRegistry0.7.0 >>> >>> The vote will be open for 72 hours. >>> Please download the release candidate and evaluate the necessary items >>> including checking hashes, signatures, build from source, and test. Then >>> please vote: >>> >>> [ ] +1 Release this package as nifi-registry-0.7.0 >>> [ ] +0 no opinion >>> [ ] -1 Do not release this package because... >>> >>
Re: Jira contributor access
I added you to the contributors role. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 15, 2020, at 9:26 AM, Bora Avcı wrote: > > jira username : boraavci > > Thanks. > > Bora AVCI
Re: Request JIRA contributor access
I’ve added you to the contributors. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 10, 2020, at 10:35 PM, Cuong Nguyen wrote: > > Hi there, > > I'd like to request for JIRA contributor access on the Nifi project. > My Github username: ndc629 > > Thanks in advance, > Cuong Nguyen.
Re: [DISCUSS] rename master branch, look through code for other related issues
I will make a Jira for the key management references. I have already removed a large portion of the legacy terms from the proxy path handling and related documentation in [1]. [1] https://github.com/apache/nifi/pull/4351 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 6, 2020, at 8:21 PM, Tony Kurc wrote: > > After a quick review, in nifi, I wasn't able to find references to the > properties Joe mentioned, but I did use crude recursive greps to look. A > majority of the blacklist references were in wali in method names and log > messages. Picking a more useful term in wali would probably make sense. > I'll make a jira, but we'd need to be a bit more deliberate about when that > change could happen, no? Long story short, since assumptions were maybe a > bit off (i.e. not an additive change), I think a later release may make > sense. > > There was a blacklist property in nifi-cpp. I'll make a jira and work on a > pr for that. > > Most prevalent in my grep analysis related to master/slave were carrying > over terms from dependencies (e.g. MySQL, zookeeper, or third-party libs in > nifi-cpp) and "master key" crypt related stuff. > > On Mon, Jul 6, 2020, 6:02 PM Tony Kurc wrote: > >> Mike, >> I did a quick check to see if anyone had done a jira or pr for #2, so I'll >> take a stab at doing that today. >> >> Tony >> >> On Thu, Jul 2, 2020 at 10:27 PM Mike Thomsen >> wrote: >> >>> Didn't see any commits or PRs for any of these yet. Do we want to consider >>> these blockers for 1.12 or hold off until a post 1.12 release? >>> >>> On Mon, Jun 22, 2020 at 12:48 PM Joe Witt wrote: >>> >>>> Not that I'm aware of. All this so far just looks really easy to deal >>>> with. >>>> >>>> On Mon, Jun 22, 2020 at 9:46 AM Mike Thomsen >>>> wrote: >>>> >>>>> Out of curiosity... are there any cases you've found where we might >>> have >>>> a >>>>> term misalignment with what another product calls them? Like we might >>>> have >>>>> primary/replica and the supported system uses master/slave? >>>>> >>>>> On Mon, Jun 22, 2020 at 11:32 AM Joe Witt wrote: >>>>> >>>>>> ...additional note after reviewing the presence of >>>> 'whitelist/blacklist' >>>>> I >>>>>> remain of the view what we need to do here is easy. There is >>> minimal >>>> API >>>>>> impact and it appears to be just the nifi.properties file for a >>>> property. >>>>>> Other code changes do not appear to be API related and seem fair >>> game >>>>> now. >>>>>> We can easily support the old naming and create a different property >>>> name >>>>>> for the properties file case. We dont need to wait for any major >>>> release >>>>>> as far as I can tell >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Sat, Jun 20, 2020 at 4:47 PM Mike Thomsen < >>> mikerthom...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I think just shooting for #1 right away makes sense, but #2 will >>> need >>>>> to >>>>>> be >>>>>>> done as part of a major release. I think we go all-in to be >>>> consistent >>>>> on >>>>>>> allow/block vs white/black and similar changes that are needed. We >>>>> should >>>>>>> also avoid things like the proposal to use "allowlist/denylist" >>> that >>>>>> other >>>>>>> teams are debating since that is just a pointless spawning of >>>>> neologisms >>>>>>> for the sake of creating them. The best approach is to use clear, >>>>> concise >>>>>>> language that is preferably as limited on jargon as possible, and >>> I >>>>> feel >>>>>>> like those teams are missing the mark on that. If we do find >>> language >>>>>> that >>>>>>> needs to be changed in descriptor name fields, I think it would >>> also >>>>>>> prevent any problems by making part of the messaging being that >>> those >>>>>>> changes are non-negotiable as
Re: [nifi] branch main created (now 239a2e8)
The PRs should be mergeable to main automatically as I _moved_ the branch rather than copying it, so unless there are new code conflicts, there shouldn’t be a problem. There are tools from other developers [1] which can automate the process of rebasing, but in general I don’t think this is necessary. For anyone who wishes to relabel their local working copy branches, you can follow these instructions from [2]. If someone has a local clone, then can update their locals <https://twitter.com/xunit/status/1269881005877256192> like this: $ git checkout master $ git branch -m master main $ git fetch $ git branch --unset-upstream $ git branch -u origin/main $ git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main I will send new updates when the other NiFi subprojects (Registry, MiNiFi Java, MiNiFi C++, and FDS) have had their default branches renamed. [1] https://github.com/ethomson/retarget_prs <https://github.com/ethomson/retarget_prs> [2] https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx <https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 7, 2020, at 9:14 AM, Matt Burgess wrote: > > IMO that's a yes, the branch is created and now main has more commits > than master and is the "source of truth". We should be using main as > the default branch (the way it is in Github now), rebasing PRs atop > main and changing the PRs in Github to have main as the base branch > (so no mistakes are made when merging). With any luck master just kind > of fades away, or perhaps at some point we just delete it. > > With some scripting we should be able to automate such PR updates, but > personally it felt a little heavy-handed so I haven't written said > script(s). Instead I'm just updating my own PRs and when I review > someone's, I'm leaving a comment to please switch over to main as the > base branch (both in the code and the PR). > > That's just my two cents though, happy to discuss further. I think we > really just need to avoid updating the master branch from here on out > rather than trying to keep them in sync, or even worse having them > diverge each with new commits that aren't in the other. > > Regards, > Matt > > On Tue, Jul 7, 2020 at 12:06 PM Otto Fowler wrote: >> >> So, is that a yes? >> >> On July 7, 2020 at 10:53:59, Matt Burgess (mattyb...@apache.org) wrote: >> >> Thanks Mike! I rebased against main since that's our default branch >> from now on, so should be ok now. >> >> On Tue, Jul 7, 2020 at 10:52 AM Mike Thomsen >> wrote: >>> >>> I'll take it. >>> >>> On Tue, Jul 7, 2020 at 9:57 AM Matt Burgess wrote: >>> >>>> I've got a PR [1] that's approaching 1.5 years since the last update, >>>> but it should be good to go and fairly straightforward to test, any >>>> takers? :) I just rebased it against the laster master branch, should >>>> I be using main as of this point, or is there more work to be done >>>> before we switch over? >>>> >>>> Thanks, >>>> Matt >>>> >>>> [1] https://github.com/apache/nifi/pull/2718 >>>> >>>> On Tue, Jul 7, 2020 at 5:48 AM Mike Thomsen >>>> wrote: >>>>> >>>>> Might also be a good time to think about closing out old PRs (like >> age > >>>>> 1.5 years). >>>>> >>>>> On Mon, Jul 6, 2020 at 10:49 PM Otto Fowler >>>> wrote: >>>>> >>>>>> So, should we rebase our outstanding PR’s? Should all outstanding >>>> PR’s >>>>>> get a note? >>>>>> >>>>>> On July 6, 2020 at 17:57:25, Andy LoPresto (alopre...@apache.org) >>>> wrote: >>>>>> >>>>>> Apache Infra completed the change. Everything looks ok from my end >> but >>>> as I >>>>>> made the initial change, can someone else verify that their local >> repo >>>> is >>>>>> good to go? Once that’s done, I’ll update the committer guide and >>>> GitHub PR >>>>>> template with the new language, and check the CI/CD for GitHub >> Actions >>>> too. >>>>>> Thanks. br/> < >>>>>> >>>>>> Andy LoPresto >>>>>> alopre...@apache.org >>>>>> alopresto.apa...@gmail.com >>>>>> He/
Re: [nifi] branch main created (now 239a2e8)
Apache Infra completed the change. Everything looks ok from my end but as I made the initial change, can someone else verify that their local repo is good to go? Once that’s done, I’ll update the committer guide and GitHub PR template with the new language, and check the CI/CD for GitHub Actions too. Thanks. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 6, 2020, at 11:02 AM, Joe Witt wrote: > > Looking into this we might be fine. We use the github 'default branch' > which I think you'll get changed by the INFRA ticket. > > We then might want to look at https://github.com/ethomson/retarget_prs > > Thanks > > On Mon, Jul 6, 2020 at 9:21 AM Joe Witt wrote: > >> Andy >> >> I *think* we need to change the github actions config a little bit as >> well. It defaults to master I believe and we can specify an alternative >> branch of 'main'. Maybe we just keep an eye on it and see if there is a >> material difference. One example is the 'build passing' indicator comes >> from master unless you override it (as I've read previously). >> >> Thanks >> >> On Mon, Jul 6, 2020 at 9:18 AM Andy LoPresto wrote: >> >>> It appears that we don’t have the ability to switch the default branch >>> ourselves; I’ve filed https://issues.apache.org/jira/browse/INFRA-20487 < >>> https://issues.apache.org/jira/browse/INFRA-20487> requesting Apache >>> Infra takes care of that. Until that time, committers _should_ merge the >>> code to main but I will monitor to ensure we keep the branches in sync. The >>> history was moved, the only remaining step is GitHub being aware of the >>> default branch change. >>> >>> >>> Andy LoPresto >>> alopre...@apache.org >>> alopresto.apa...@gmail.com >>> He/Him >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>>> On Jul 4, 2020, at 11:57 AM, Joey Frazee >>> wrote: >>>> >>>> @alopresto Do you want everyone individually pushing to both master and >>> main or are you intending to keep it in sync until it’s made the default? >>>> >>>> -joey >>>> On Jul 2, 2020, 6:39 PM -0500, alopre...@apache.org, wrote: >>>>> This is an automated email from the ASF dual-hosted git repository. >>>>> >>>>> alopresto pushed a change to branch main >>>>> in repository https://gitbox.apache.org/repos/asf/nifi.git. >>>>> >>>>> >>>>> at 239a2e8 NIFI-7513 Added custom DNS resolution steps to walkthrough >>> (#4359) >>>>> >>>>> No new revisions were added by this update. >>>>> >>> >>>
Re: [nifi] branch main created (now 239a2e8)
It appears that we don’t have the ability to switch the default branch ourselves; I’ve filed https://issues.apache.org/jira/browse/INFRA-20487 <https://issues.apache.org/jira/browse/INFRA-20487> requesting Apache Infra takes care of that. Until that time, committers _should_ merge the code to main but I will monitor to ensure we keep the branches in sync. The history was moved, the only remaining step is GitHub being aware of the default branch change. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 4, 2020, at 11:57 AM, Joey Frazee > wrote: > > @alopresto Do you want everyone individually pushing to both master and main > or are you intending to keep it in sync until it’s made the default? > > -joey > On Jul 2, 2020, 6:39 PM -0500, alopre...@apache.org, wrote: >> This is an automated email from the ASF dual-hosted git repository. >> >> alopresto pushed a change to branch main >> in repository https://gitbox.apache.org/repos/asf/nifi.git. >> >> >> at 239a2e8 NIFI-7513 Added custom DNS resolution steps to walkthrough (#4359) >> >> No new revisions were added by this update. >>
Re: [discuss] 1.12.0 or 1.11.5...
Martin, I understand everyone is anxious to get their hands on the next release, but this thread is exactly how we determine what will be in it. There is an inherent balance in determining which features and fixes need to be included to make the release worth it and which will delay it, and this requires being aware of a large percentage of the community’s needs. In addition, once a committer has volunteered as release manager (which I believe Joe has already done for this release), they set the pace and will solicit feedback from the community here. So if Joe replies that he wants to perform the release process on Monday, any non-critical tickets (i.e. features, non-security bug fixes, etc.) which are not merged will not go into 1.12.0. However, we often do wait to perform the release process until specifically enumerated features are reviewed and merged so they can be included. One way to encourage the timely release of the next version is to offer code reviews and other community activities where possible, because that helps move everything forward. In the early days of the project, the user base and the contributor base overlapped highly, and it was easier to solicit reviews on all contributions because the majority of the other active people were also developers. Over the last 5 years, both the contributor base and the user base have grown substantially, but the user base has grown far faster than the contributor base. For this reason, we have many more people asking for features, releases, etc. than can contribute them. Part of being a community is helping where possible, with the understanding that not everyone will have the time, expertise, or desire to do all tasks. But the most constructive way to speed the release process is for the outstanding tickets whose inclusion is determined to be necessary to be reviewed and merged. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 2, 2020, at 7:09 PM, Mike Thomsen wrote: > > ** that's not to say I'll try to hold up a release vote over it, but there > is a good reason behind asking that it be included in the last round of > reviews. > > On Thu, Jul 2, 2020 at 10:07 PM Mike Thomsen wrote: > >> #4217 is something I need because I'm planning to build a Cassandra-based >> DistributedMapCache service for my client and contribute it back once we >> determine that it's stable enough to give back. >> >> On Thu, Jul 2, 2020 at 9:26 PM Martin Ebert wrote: >> >>> Hi Mike, >>> I suggest we wrap this up and only include your PRs in version 1.13 Are >>> there any objections to release NiFi 1.12? We should not keep the >>> community >>> waiting any longer. >>> >>> Best, >>> Martin >>> >>> Mike Thomsen schrieb am Mi., 10. Juni 2020, >>> 00:14: >>> >>>> I would like to refactor the OAuth2TokenProvider controller service to >>> use >>>> the class that InvokeHttp now delegates to (OkHttpClientBuilderUtils?) >>> for >>>> SSL configuration. I could get a pretty fast turn around on that one. >>>> >>>> Also, would like to see if we can get >>>> https://github.com/apache/nifi/pull/4217 and >>>> https://github.com/apache/nifi/pull/4204 reviewed and merged. >>>> >>>> The latter is a nice-to-have that can wait until 1.13 if need be. >>>> >>>> On Tue, Jun 9, 2020 at 1:32 PM Mark Payne wrote: >>>> >>>>> Joe, >>>>> >>>>> I do think there are a couple of things that I’d like to see make it >>> into >>>>> 1.12. >>>>> I worked on NIFI-7476 [1] and it’s been merged to master. But it adds >>> a >>>>> couple of fields to the data model for Process Groups. So that, >>>>> unfortunately, means we need to update the data model in NiFi Registry >>>> [3] >>>>> and release that before releasing NiFi. Otherwise, those changes can >>> be >>>>> made in NiFi but when a group pushed to the registry, the changes >>> will be >>>>> lost. I’m working NIFIREG-398 now and should have a PR up soon. Also, >>>>> related is NIFI-7509 [4], which I have a PR up for. >>>>> >>>>> I am hoping to have all of this merged by end of the week. >>>>> >>>>> Thanks >>>>> -Mark >>>>> >>>>> >>>>> [1] https://issues.apache.org/jira/browse/NIFI-7476 >>>>> [2] https://issues.apache.org/jira/browse/NIFI-7509 &g
Re: Next release NiFi 1.12
Here is the discussion thread [1]. [1] https://lists.apache.org/thread.html/r2cfeda59f96613c9d1739f4f2a1fd83ab15a4ff77d4659154a45bcac%40%3Cdev.nifi.apache.org%3E <https://lists.apache.org/thread.html/r2cfeda59f96613c9d1739f4f2a1fd83ab15a4ff77d4659154a45bcac@%3Cdev.nifi.apache.org%3E> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jul 2, 2020, at 11:32 AM, Joe Witt wrote: > > Martin > > Youll need to subscribe to dev to see this but there is already a discuss > thread for it. As I mentioned on slack we just need to do it. There was > some items folks were trying to wrap. > > Thanks > > On Thu, Jul 2, 2020 at 6:20 AM Martin Ebert > wrote: > >> Hi community, >> there are so many great new features waiting to be made available to the >> whole community as NiFi 1.12 Are there any reasons to wait for a release? >>
Re: Need help regarding some challenges while using Apache Nifi
Shweta, if you need responses within a certain timeframe, you may want to
investigate commercial vendor support. The Apache NiFi open source community
attempts to answer any questions we can, but makes no guarantee about accuracy,
availability, or response time.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 29, 2020, at 4:24 AM, shweta soni wrote:
>
> Hello Team,
>
> Request you to please help me with the issues mentioned in the trailing
> mail. Thanking you in anticipation.
>
> Thanks & Regards
> Shweta Soni
>
> Am Do., 25. Juni 2020 um 23:36 Uhr schrieb shweta soni :
>
>> Hello Team,
>>
>>
>>
>> We are using Nifi in our data ingestion process. The version details are:
>> *Nifi
>> 1.11.4 , Cloudera Enterprise 5.16.2 and Hive 1.1*. I posted my issues on
>> Nifi SLACK channel, but did not get answer for some of my questions. So I
>> am posting all my queries here in ardent hope of getting
>> solutions/workarounds for them. We are facing below issues:
>>
>>
>>
>>
>>
>> 1. *SCENARIO*: In RDBMS Source we have Date/Timestamp column. In
>> Hive Destination we have Date/Timestamp columns but when we are trying to
>> ingest from source to destination, we are getting Int/Longwritable cannot
>> be written to Date/Timestamp errors in Hue. We are using following
>> processors: QueryDatabaseProcessor à UpdateRecord( column mapping and
>> output schema) à PutHDFS à ReplaceText à PutHiveQL. Below are the Avro
>> Output Schema since we don’t have Date or Timestamp as datatype in Avro
>> Schema.
>>
>>
>> {"name":"dob","type":["null",{"type":"long","logicalType":"timestamp-millis"}
>>
>>{"name":"doA","type":["null",{"type":"int","logicalType":"date"}
>>
>>
>> *Q. Please let me know how can we put data/timestamp source columns to
>> data/timestamp destination columns?*
>>
>>
>>
>>
>>
>>
>> 2. *SCENARIO* : Decimal data is not being inserted in ORC table.
>>
>> Solution: I am loading data in Avro table and then doing INSERT INTO ORC
>> table from it. This solution I found from Cloudera community.
>>
>>
>> *Q. Is there any other solution for loading decimal data in ORC table?*
>>
>>
>>
>>
>>
>>
>> *3. **SCENARIO**: *We have a 1 time full load flow in Nifi –
>> QueryDatabase à PutHiveQL. à LogAttribute. This acts as a pipeline in our
>> custom based UI. This will run only once. In Nifi UI we can manually start
>> processors to start the flow and once all the flowfiles are processed and
>> the success queue of PutHiveQL becomes empty we can stop the processor in
>> Nifi UI. But now we want to know programmatically that this flow ended at
>> particular time and we want to show the pipeline status as completed in our
>> custom based UI. So how can we stimulate this?
>>
>>
>>
>> *Q. *Since Nifi is for continuous data transfer, how can we know
>> that a particular flow has ended?
>>
>>
>>
>>
>>
>>
>> 4. *SCENARIO** :*I have Hive table with complex datatypes i.e. Array,
>> Map. When I am trying to get this data via SELECTHIVEQL processor , it is
>> giving output in String format for all the columns. Then in next
>> UpdateProcessor it is giving error that string datatype cannot be converted
>> to Array or Map.
>>
>> Avro Output Schema:
>>
>> {"type": "array", "items": "double"}
>>
>> {"type": "map", "values": "int"}
>>
>>
>> *Q. How to handle complex datatype in Hive via Nifi. Source table as Hive
>> and destination table as another Hive table.*
>>
>>
>>
>>
>>
>>
>> 5. *SCENARIO*: In QueryDatabase Processor we have Max-value column
>> which helps in incremental load. But there is not such functionality for
>> Hive table incremental load(i.e.SELECTHIVEQL). I tried with
>> GenerateTableFetch processor and QueryDatabase processor using Hive1_1
>> Connection service but it is not working. I was told on Nifi SLACK channel
>> to raise JIRA for new processor GenerateHiveTableFetch/QueryHiveDatabase
>> processor.
>>
>>
>>
>> *Q. Is there any alternative in which we can handle hive table incremental
>> load or should I go ahead and raise JIRA for the same?*
>>
>>
>>
>> Request you to please help us. Thanking you in anticipation.
>>
>>
>>
>>
>>
>> Thanks & Regards,
>> Shweta Soni
>>
Re: Jira Contribution Access.
Hi YoungGyu, I have added you to the contributors role on Jira. You can now assign tickets to yourself. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 26, 2020, at 8:40 AM, younggyu Chun wrote: > > I'd like to contribute to https://issues.apache.org/jira/browse/NIFI-7579 and > assign me this ticket. My user name is younggyuchun. > > Thank you, > younggyu
Re: NiFi PriorityAttributePrioritizer doesn't prioritize all items in a queue.
Hi Ryan, Thanks for writing such a detailed report. What version of NiFi are you observing this behavior against? I know there were some issues in older versions with queue swapping, but since you linked to current code, I’m assuming you’re experiencing this on 1.11.4? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 24, 2020, at 11:01 AM, Ryan Hendrickson > wrote: > > Hello, > We've noticed that the PriorityAttributePrioritizer doesn't prioritize > all items in a queue. > > We'll get a queue of 50,000 items in it, then a 50,001 item will be > added with a #1 priority. This item won't be prioritized above lower > priority items in the queue. > > We did some research into this and wanted to confirm what we found... > >For any Relationship there are 2 underlying queues: > (1) An Active Queue (java.util.PriorityQueue) for the first 20,000 > items, defined by the nifi.queue.swap.threshold in nifi.properties. > (2) A Swap Queue (java.util.ArrayList) for the rest of the > queue's items. > >If the Active Queue is full, every new item, regardless of priority, is > placed on the Swap Queue. No item on the Swap Queue will be re-prioritized > until the entire Active Queue is empty (Line 284 > <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/SwappablePriorityQueue.java#L284>). > Thus, the PriorityAttributePrioritize only "actively" sorts the first > 20,000 items on the relationship. Once the queue is empty, it'll re-sort > the rest and move the top 20,000 over. Then the problem repeats. > >I hadn't read anywhere in the documentation that this is the case. > Even in the admin guide definition of the nifi.queue.swap.threshold it > doesn't mention that it affects the PriorityQueue system. We just raised > it from 20,000 to 200,000 but we're all assuming that's going to have > detrimental effects elsewhere. > > If you guys could confirm our suspicions here, we'd appreciate that. > Also any suggestions on what to do here, or how an increased threshold size > negatively impacts the JVM. > > > Stack we followed to figure this out: > > SwappablePriorityQueue.java: doPoll() > SwappablePriorityQueue.java: poll() > StandardFlowFileQueue.java: queue.poll() > StandardConnection.java flowFileQueue.poll() > StandardProcessSession.java: conn.poll() > InvokeHttp.java: session.get() > > Thanks, > Ryan
Re: Jira contributor access
Hi Javi, I have added you to the contributors role so you can assign tickets to yourself. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 24, 2020, at 9:29 AM, Javi Roman wrote: > > I would like to receive contributor access to Jira. > My username is "javiroman". > > I'd like to assign me the task: > https://issues.apache.org/jira/browse/NIFI-7578 > > The code is done, working in the patch right now (if this feature has sense > for the project). > > -- > Javi Roman > > Twitter: @javiromanrh > GitHub: github.com/javiroman > Linkedin: es.linkedin.com/in/javiroman > Big Data Blog: dataintensive.info > Apache Id: javiroman
Re: [DISCUSS] rename master branch, look through code for other related issues
Hi Edward, thanks for sharing your thoughts. I’ll reply inline. > - Some of the terms proposed are not industry standard and may potentially > cause significant issue for non-english speakers. I actually believe making these changes will _improve_ the clarity for non-english speakers. “Whitelist” and “blacklist” confer no inherent reason to mean allow and deny other than connotative biases. “Allow” and “deny” explicitly indicate the verb that is happening. Another example is branch naming. “Masters” don’t have “branches”. “Trunks” do. These terms make _more_ sense for a non-English speaker than the current terms. > > - For each change that is made can we guarantee that we will not lose > clarity of meaning, and then have revert the change down the line if the > change causes a drop in usage. I don’t expect the community will opt to change the new terms back to ones with negative connotations in the future. If there is discussion about it, this thread will provide good historical context for why the decision was made to change it, just as the mailing list discussions do for other code changes. > > - Of what percentage of people is this truly an issue for and what > percentage isn't. Any change that has the potential to cause a major split > in the community, there must be as close as possible to a majority, and not > just from those that are vocal and active on the mailing lists. > Disscustions on other groups are turning toxic, and in some cases are > potentially leading to the collapse of these projects where these changes > are being implemented with what appears to be without the agreement of a > signifficant chunk of the community. > In my perspective this should be an issue for the entire community. Being able to identify an issue that directly affects another person but not one’s self is the definition of privilege. If I can look at how the use of these words in someone’s daily life or career impacts them negatively, when the change would not harm me at all, I see that as a failure on my part. I understand the desire to hear from the silent majority, but active participation and discussion on the mailing list is the exact measure described by the Apache process for participation in the community. Those who speak here are the ones who will have a voice. > - From a personal perspective, I sit on the autism spectrum and have grown > up with people using words that are very offensive and have hurt me badly. > Instead of having these words as offensive and untouchable. Myself and > others have instead made these words our own and made them lose the > negative connotations they have. As such, I do find the current > disscustions deeply alarming and feels like they start to border into the > realm of censorship. > I think it’s admirable that you have responded to negative circumstances in that way. I also recognize that not everyone has that opportunity. If we can take these actions as a community to improve the experience for others, I am in favor of that. > - One final point (and potentially controversial), A good chunk of the > wording that is proposed to be changed. Is being done so on the > "modern"/"street" definition of these words and not the actual definition. > Language should change and evolve to introduce clarity, but right now does > this change improve the clarity across the engineering sector and I believe > it won't. I’ll paraphrase Emily Kager here with “developers spend an inordinate amount of time and energy arguing about the meaning and semantics of variable and method names, but pretend exclusionary terms are meaningless.” [1] If we can expend that much energy deciding if a method creates vs. builds vs. forms an imaginary concept like a LibraryFrameworkWrapperDecorator, I refuse to concede that we can and in fact should do so with the terms that actually affect our community members’ lives. [1] https://twitter.com/EmilyKager/status/1271102865889734656 <https://twitter.com/EmilyKager/status/1271102865889734656> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 17, 2020, at 6:43 PM, Edward Armes wrote: > > This is a difficult issue and causes no small amount of friction every > time. I'm personally against this for the following reassons: > > - Some of the terms proposed are not industry standard and may potentially > cause significant issue for non-english speakers. > > - For each change that is made can we guarantee that we will not lose > clarity of meaning, and then have revert the change down the line if the > change causes a drop in usage. > > - Of what percentage of people is this truly an issue for and what > percentage isn't. Any change that has the potential to caus
Re: [DISCUSS] rename master branch, look through code for other related issues
I am a proponent of making this change and also using allow/deny list, meddler-in-the-middle, etc. Here is a blog [1] with easy instructions for executing the change in git, although I don’t know if there is any Apache-integration specific changes we would also need. [1] https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 17, 2020, at 3:06 PM, Joe Witt wrote: > > I suspect it would be fairly easy to make this change. We do, I think, > have whitelist/blacklist in there somewhere but im not sure how involved. > > On Wed, Jun 17, 2020 at 3:04 PM Tony Kurc wrote: > >> All, >> I've seen the discussion started on other projects [1][2], so I wanted to >> kick off a discussion to determine whether this is something nifi could >> look at too. Allen Wittenauer's post to yetus captures the why and some of >> the how, so rather than copy and pasting, you can take a look at what he's >> done. Thoughts? >> >> Tony >> >> 1. >> >> https://lists.apache.org/thread.html/rd38afa9fb6c0dcd77d1a677f1152b7398b3bda93c9106b3393149d10%40%3Cdev.yetus.apache.org%3E >> 2. >> >> https://lists.apache.org/thread.html/r0825eec0c84296bdab7cf898a987f06355443241ca02b2aaa51d3ef9%40%3Cdev.accumulo.apache.org%3E >>
Re: In memoriam of Jeff Storck
Jeff loved mocks, both friendly impressions and in his tests. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 15, 2020, at 12:12 PM, Jeremy Dyer wrote: > > This is shocking and heartbreaking news. Jeff was a great guy and will be > deeply missed. > > The last time I saw Jeff in person was with Aldrin. We were eating at Bonchon > chicken and he was mocking me for how little spice I could handle XD. I could > always count on him for a good Dumb and Dumber reference and laugh. We also > shared a common hatred for conference food. > > RIP Jeff > > On Mon, Jun 15, 2020 at 2:33 PM Joe Witt <mailto:joe.w...@gmail.com>> wrote: > You will be greatly missed. Your impact to this community has been > tremendous. The items Andy summarizes were huge efforts that you drove over > periods of many many months if not a year or more and they make NiFi so much > more accessible than before. > > RIP Jeff. > > > > On Mon, Jun 15, 2020 at 11:24 AM Andy LoPresto <mailto:alopre...@apache.org>> wrote: > It is with a heavy heart that I write to the NiFi community today. Jeff > Storck, a PMC member, committer, and genuine and helpful presence in the > community, has passed away. > > I was lucky enough to know Jeff personally for many years, and his absence is > a huge loss to all of us who did. Jeff was incredibly intelligent, but also > kind and willing to share his experience with everyone. Whether playing > volleyball (I am nowhere near as good but he humored me), discussing the best > ramen and sushi spots, or evaluating a new exercise regime, Jeff brought > passion to everything. A number of us are sharing stories of our favorite > times with Jeff, and I am touched by how many people have a memory of Jeff > reaching out and patiently helping them when they were new or struggling with > a task. > > While other colleagues would happily transition to any topic _but_ work when > we went to a nearby brewery at the end of a long day, Jeff would sit down > next to me and say with a smile, "Ok Andy, work's done, now we can _really_ > talk about Groovy unit testing." He never shied away from expressing his > perspective and stood on conviction, but he was also open and genuinely > wanted to hear other views to expand his mind. > > If you come across a Spock test in the NiFi codebase, that was most likely > Jeff's work. He was intimately involved in much of the most challenging code > - especially Kerberos integration, making the difficult but critical > processes easier for our users. Anyone running NiFi on Java 11 should thank > Jeff, as that was a labor of love, pushing against the headwinds of so many > compatibility issues and language changes. The ease with which NiFi runs on > multiple versions and platforms belies the immense amount of effort and > dedication that he put into making this happen. > > There are so many aspects to Jeff that a note like this could never capture, > but one that stands above the rest to me is Jeff's passion for learning and > growth. He devoted himself to doing the best he could and constantly > improving that. That is a noble philosophy that I know I will remember and > admire moving forward. I’ve already started learning Kotlin because of Jeff’s > enthusiasm and encouragement. > > Jeff’s family has created a GoFundMe page [1] and there they describe their > intent to celebrate his life. I think that message is very positive and > uplifting. To anyone wondering how they can honor Jeff's legacy, I suggest > offering a helping hand to someone who needs it. Something as simple as > responding to an extra "newbie" mailing list question at the end of a long > day, or taking on a challenging task because your neighbor has their plate > full. That's how Jeff lived, and he made the world a better place. > > > > Andy > > [1] https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck > <https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck> > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >
Re: secure Nifi with ingress configuration
Please describe the error you are encountering and what you expect to happen.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 14, 2020, at 9:59 PM, Ganesh, B (Nokia - IN/Bangalore)
> wrote:
>
> Hi ,
>
> Could you please help me to unblock .
>
> Thanks & Regards,
> Ganesh.B
>
> -Original Message-
> From: Ganesh, B (Nokia - IN/Bangalore)
> Sent: Friday, June 12, 2020 11:05 AM
> To: dev@nifi.apache.org
> Subject: secure Nifi with ingress configuration
>
> Hi ,
>
> I am trying to configure secured nifi with ingress , I have not succeed could
> you please help me to correct the configuration please .
>
> Installed citm-ingress and then flowed procedure
>
> Created the secret by using .pem file generated by below procedure and
> created k8s secret
>
> • openssl pkcs12 -in CN=admin_OU=NIFI.p12 -passin
> pass:uva1qK6Rlgw3wSM8dP344oMr5DzjET9Sjpj6TSkcpTU -out test.pem
>
> • openssl x509 -in test.pem -out test.crt
>
> • openssl rsa -in test.pem -out test.key
>
> • kubectl create secret generic ingress-tls-certificate
> --from-file=tls.crt=test.crt --from-file=tls.key=test.key
>
> ingress.yaml
>
> apiVersion: extensions/v1beta1
> kind: Ingress
> metadata:
> name: "{{ template "nifi.fullname" . }}-gui-ingress"
> annotations:
>nginx.ingress.kubernetes.io/proxy-redirect: "default"
>nginx.ingress.kubernetes.io/rewrite-target: /$2
>nginx.ingress.kubernetes.io/secure-backends: "true"
>nginx.ingress.kubernetes.io/ssl-passthrough: "false"
>nginx.ingress.kubernetes.io/ssl-redirect: "true"
>ingress.citm.nokia.com/sticky-route-services:
> "$cookie_JSESSIONID|JSESSIONID ip_cookie"
>nginx.ingress.kubernetes.io/configuration-snippet: "set_cookie_flag *
> secure;"
>nginx.ingress.kubernetes.io/proxy-body-size: "1m"
> labels:
>chart: {{ .Chart.Name }}
>release: {{ .Release.Name }}
>heritage: {{ .Release.Service }}
> spec:
> rules:
> - http:
> paths:
> - backend:
> serviceName: {{ template "nifi.fullname" . }}-gui-ingress
> servicePort: 9443
>path: /test(/|$)(.*)
> tls:
>- secretName: ingress-tls-certificate
>
> Thanks & Regards,
> Ganesh.B
>
Re: asciidoctor-maven-plugin error on windows
Vasily, This is not a known issue, as every PR runs a Windows Java 8 build which includes this step. If you do file a Jira, please provide details on your Maven version & configuration, OS version, full output of logs & errors, etc. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 11, 2020, at 5:16 AM, Vasily Makarov > wrote: > > Hi all! > When building NiFi on windows I encounter such an error with > asciidoctor-maven-plugin: > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:22 min (Wall Clock) > [INFO] Finished at: 2020-06-11T13:42:20+03:00 > [INFO] > > [ERROR] Failed to execute goal > org.asciidoctor:asciidoctor-maven-plugin:1.5.2:process-asciidoc (output-html) > on project nifi-docs: Execution output-html of goal > org.asciidoctor:asciidoctor-maven-plugin:1.5.2:process-asciidoc failed: > (LoadError) no such file to load -- asciidoctor -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :nifi-docs > > > > I have temporarily resolved this issue by commenting out these rows in > nifi-docs pom.xml: > > > > > > > > > > > > > Building with Java 1.8.0_211-b12, but I also have tried with newer versions > of Java. Is it a known issue or should I create an issue in Jira?
Re: Build error on nifi-site-to-site-client
Hi Mark, I just worked on these areas and those tests were passing successfully for me on Mac OS X 10.15.5 with OpenJDK 11.0.6 and Maven 3.6.3. They also passed on the GitHub Actions builds using Java 11. Do you have any custom DNS configurations or certificates imported in your OS keystore which might be affecting this? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 10, 2020, at 10:22 AM, Mark Bean wrote: > > I am attempting to build master (01e42dfb) using Java 11, but getting test > failures. It builds fine with Java 8. There are two failing tests with > stack traces shown below. > > Anyone else experiencing similar issues? Or, are there suggestions on what > might be the problem? > > MacOS10.15.5 > openjdk-11.0.2 > Maven 3.6.3 > > [ERROR] > testSendSuccessHTTPS(org.apache.nifi.remote.client.http.TestHttpClient) > Time elapsed: 0.092 s <<< ERROR! > > java.io.IOException: Failed to complete transaction with Peer[url= > https://localhost:62046/nifi-api] due to java.io.IOException: [Url= > https://localhost:62046/nifi-api, TransferDirection=SEND, > State=TRANSACTION_CONFIRMED] Failed to receive a response from Peer[url= > https://localhost:62046/nifi-api] when expecting a TransactionFinished > Indicator. It is unknown whether or not the peer successfully > received/processed the data. javax.net.ssl.SSLPeerUnverifiedException: peer > not authenticated > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:931) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:789) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:905) > > Caused by: java.io.IOException: [Url=https://localhost:62046/nifi-api, > TransferDirection=SEND, State=TRANSACTION_CONFIRMED] Failed to receive a > response from Peer[url=https://localhost:62046/nifi-api] when expecting a > TransactionFinished Indicator. It is unknown whether or not the peer > successfully received/processed the data. > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:931) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:789) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:905) > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:931) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:789) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:905) > > > [ERROR] > testSendLargeFileHTTPS(org.apache.nifi.remote.client.http.TestHttpClient) > Time elapsed: 0.068 s <<< ERROR! > > java.io.IOException: Failed to complete transaction with Peer[url= > https://localhost:62046/nifi-api] due to java.io.IOException: [Url= > https://localhost:62046/nifi-api, TransferDirection=SEND, > State=TRANSACTION_CONFIRMED] Failed to receive a response from Peer[url= > https://localhost:62046/nifi-api] when expecting a TransactionFinished > Indicator. It is unknown whether or not the peer successfully > received/processed the data. javax.net.ssl.SSLPeerUnverifiedException: peer > not authenticated > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:931) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendLargeFile(TestHttpClient.java:944) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendLargeFileHTTPS(TestHttpClient.java:1014) > > Caused by: java.io.IOException: [Url=https://localhost:62046/nifi-api, > TransferDirection=SEND, State=TRANSACTION_CONFIRMED] Failed to receive a > response from Peer[url=https://localhost:62046/nifi-api] when expecting a > TransactionFinished Indicator. It is unknown whether or not the peer > successfully received/processed the data. > javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:931) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendLargeFile(TestHttpClient.java:944) > > at > org.apache.nifi.remote.client.http.TestHttpClient.testSendLargeFileHTTPS(TestHttpClient.java:1014) > > Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authe
Re: Potential Problems on Discarding Return Values
Hi Adam,
The mailing list strips a lot of attachments, your PDF included. The best way I
have found to share this kind of information with the community is to put it on
the Apache NiFi Wiki [1] or post a GitHub Gist [2], etc. Thanks.
[1] https://cwiki.apache.org/confluence/display/NIFI/Apache+NiFi
<https://cwiki.apache.org/confluence/display/NIFI/Apache+NiFi>
[2] https://gist.github.com/ <https://gist.github.com/>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 10, 2020, at 3:33 AM, Adam Hunyadi
> wrote:
>
> Hi,
>
> Sorry, I did not realize, that this confluence page might not be available
> for some devs.
>
> Attached the content as pdf.
>
> Thanks,
> Adam
>
> On 2020. 06. 10. 12:11, Adam Hunyadi wrote:
>> Hi,
>>
>> Upon checking some behaviours in NiFi with Tamas Palfy, we have found that
>> function signatures like this can be a frequent source of error:
>>
>> public FlowFile penalize(FlowFile flowFile) { ... }
>>
>> I documented the issue here:
>>
>> https://cloudera.atlassian.net/wiki/spaces/ENG/pages/610534373/Discarding+Return+Values+in+NiFi
>>
>> <https://cloudera.atlassian.net/wiki/spaces/ENG/pages/610534373/Discarding+Return+Values+in+NiFi>
>> It would be nice if a Java dev with some spare time could look more into the
>> issue and share their findings.
>>
>> Cheers,
>> Adam
>>
Re: [discuss] 1.12.0 or 1.11.5...
Thanks Joe. I could use a reviewer for [1], and I’m taking a look at a couple other things that have gone in recently but should be ready very soon. [1] https://github.com/apache/nifi/pull/4228 <https://github.com/apache/nifi/pull/4228> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 9, 2020, at 10:21 AM, Joe Witt wrote: > > Team, > > Took me a bit longer than planned but am back in position to help drive a > 1.12 release. > > Anything we're still trying to wrap we need to wait for? > > Thanks > > On Tue, May 19, 2020 at 6:03 AM Joe Witt wrote: > >> Ok thanks for the heads up Bence >> >> On Tue, May 19, 2020 at 8:50 AM Simon Bence >> wrote: >> >>> Hi, >>> >>> If the community decides to go on with 1.11.5, I kindly ask to include >>> NIFI-7292 in order to fix the issue reported in NIFI-7454. Thank you! >>> >>> https://issues.apache.org/jira/browse/NIFI-7454 < >>> https://issues.apache.org/jira/browse/NIFI-7454> >>> https://issues.apache.org/jira/browse/NIFI-7292 < >>> https://issues.apache.org/jira/browse/NIFI-7292> >>> >>> Regards, >>> Bence >>> >>> >>>> On 2020. May 18., at 16:18, Joe Witt wrote: >>>> >>>> Hello >>>> >>>> I will be unavailable to conduct release management now for probably the >>>> next week or two. I'll come back to this thread when able to dive in. >>>> >>>> Thanks >>>> >>>> On Wed, May 13, 2020 at 2:11 PM Joe Witt wrote: >>>> >>>>> K - sounds good. Will circle back in a few days. >>>>> >>>>> On Wed, May 13, 2020 at 1:56 PM Andy LoPresto >>>>> wrote: >>>>> >>>>>> Thanks for starting this thread Joe. I agree the back-pressure fix is >>> one >>>>>> that will help the community a lot. I am currently working on a couple >>>>>> security features which would also greatly benefit users. I’d like to >>> see >>>>>> these get in 1.12.0 if possible, especially as there are some changes >>> that >>>>>> have already gone into master which I don’t think are ready to be >>> released >>>>>> and then required to be supported. Some are already up as PRs. >>>>>> >>>>>> My preferences, ranked: >>>>>> >>>>>> 1. Wait a few days on 1.12.0, release it with the new security >>> features >>>>>> and the back-pressure fix (along with the other 170+) >>>>>> 2. Release 1.11.5 immediately for the back-pressure fix >>>>>> 3. Release 1.12.0 immediately for the back-pressure fix and the >>> security >>>>>> features have to wait for 1.13.0 >>>>>> >>>>>> Andy LoPresto >>>>>> alopre...@apache.org >>>>>> alopresto.apa...@gmail.com >>>>>> He/Him >>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>>>> >>>>>>> On May 13, 2020, at 10:52 AM, Joe Witt wrote: >>>>>>> >>>>>>> Team, >>>>>>> >>>>>>> A couple folks have recently reported slowness in the UI in the >>> recent >>>>>>> releases particularly with larger and more production style flows. >>> It >>>>>>> appears to be related to when back pressure prediction is enabled >>> [1]. >>>>>>> >>>>>>> It also appears this is already fixed [2] and on 1.12.0-SNAPSHOT. >>>>>>> >>>>>>> I'd like to produce a release which captures this. It isn't clear >>> that >>>>>>> 1.12 is ready to roll but happy to look into that path if so. There >>> are >>>>>>> already 177+ closed JIRAs on that thing!! >>>>>>> >>>>>>> Alternatively if easier I could kick out another on 1.11 which would >>> be >>>>>>> 1.11.5 and go looking opportunistically for other bits to grab as >>> well. >>>>>>> >>>>>>> Any thoughts? >>>>>>> >>>>>>> [1] https://issues.apache.org/jira/browse/NIFI-7437 >>>>>>> [2] https://issues.apache.org/jira/browse/NIFI-7087 >>>>>>> >>>>>>> Thanks >>>>>>> Joe >>>>>> >>>>>> >>> >>>
Re: Jira contributor access - feitgraph
Hi Vasiliy, Welcome to the project. I have given you the contributor role in Jira, so you can now assign tickets to yourself. Thanks for your contributions. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 8, 2020, at 8:15 AM, Vasily Makarov > wrote: > > Hi! I would like to join Apache NiFi open source community and to participate > in open source development. My login in Apache Jira is: > feitgraph > > > I would like to start with these two tasks: > https://jira.apache.org/jira/browse/NIFI-7057 > https://jira.apache.org/jira/browse/NIFI-7035 > > Thanks in advance! > Vasiliy Makarov.
Re: Policy to clear queues
Do the node identities themselves have the proper permissions as well? The
following is from the Admin Guide:
> In order to access List Queue or Delete Queue for a connection, a user
> requires permission to the "view the data" and "modify the data" policies on
> the component. In a clustered environment, all nodes must be be added to
> these policies as well, as a user request could be replicated through any
> node in the cluster.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 4, 2020, at 4:45 PM, Phil H wrote:
>
> The action is available in the menu. I get the following pop up:
>
> *Insufficient Permissions*
>
> *Node nifiX.domain.com:443 <http://nifiX.domain.com:443> is unable to
> fulfil this request due to: Unable to modify the data for Processor with ID
> {guid}. Contact the system administrator. Contact the system administrator.*
>
> The nifi-user.log just shows successful authentication events for the user
> in question (the system is locked down to authorized users)
>
> Phil
>
> On Fri, 5 Jun 2020 at 09:25, Andy LoPresto wrote:
>
>> Are you seeing this behavior exhibited as the action is not even available
>> to those users, or when they try to execute it, it returns an error? Can
>> you examine the logs/nifi-user.log file to see if the authorization is
>> occurring successfully?
>>
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>>> On Jun 4, 2020, at 4:22 PM, Phil H wrote:
>>>
>>> Hi guys,
>>>
>>> So I checked this morning, and the users are members of a group that has
>>> “modify the data” permission at the root level (and is inherited within
>> the
>>> process group). They can start/stop processors, but cannot empty (or even
>>> list) the queues in said process group.
>>>
>>> I also set up a queue at the root level and confirmed the same behaviour
>>> there.
>>>
>>> Thanks
>>>
>>> On Thu, 4 Jun 2020 at 23:22, Bryan Bende wrote:
>>>
>>>> Would also add that if you don't have specific component policies on
>>>> processors, it should inherit from the process group. So at the process
>>>> group level you can give some users write to the actual process group
>> which
>>>> should control creating/deleting connections, and give some users only
>>>> modify the data on the process group which would control clearing
>> queues.
>>>>
>>>> On Thu, Jun 4, 2020 at 8:55 AM Mark Bean wrote:
>>>>
>>>>> Phil,
>>>>>
>>>>> There is a 'modify the data' Component Access Policy. Use the key icon
>> in
>>>>> the Operate palette (or right-click on the component) to access the
>>>>> Component Access Policies as opposed to using the Global Menu in the
>>>> upper
>>>>> right to access Global Access Policies.
>>>>>
>>>>> The user will be able to empty a queue if they are in the 'modify the
>>>> data'
>>>>> policy for the upstream component (processor) which generated the data.
>>>>> This policy does not allow the user to delete the connection between
>>>>> processors. To do so requires the 'modify the component' policy.
>>>>>
>>>>> One additional nuance to consider: if you are operating a NiFi Cluster,
>>>> you
>>>>> will need to add each of the cluster nodes to the 'modify the data'
>>>> policy
>>>>> as well. This is required because the request to empty a queue is
>> proxied
>>>>> from the node being used to access the UI out to the remaining nodes.
>>>>>
>>>>> -Mark
>>>>>
>>>>>
>>>>> On Thu, Jun 4, 2020 at 6:52 AM Phil H wrote:
>>>>>
>>>>>> Hi Andy,
>>>>>>
>>>>>> Thanks for your reply. I don’t recall seeing the modify data policy in
>>>>> the
>>>>>> user interface. Is it possible this is something I would have to
>> change
>>>>> at
>>>>>> the back end?
>>>>>>
>>>>>> I don’t have the system in front of me now, will have to confirm
>>>>> tomorrow.
>>>>>>
>>>>>> Regards,
>&g
Re: Policy to clear queues
Are you seeing this behavior exhibited as the action is not even available to those users, or when they try to execute it, it returns an error? Can you examine the logs/nifi-user.log file to see if the authorization is occurring successfully? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 4, 2020, at 4:22 PM, Phil H wrote: > > Hi guys, > > So I checked this morning, and the users are members of a group that has > “modify the data” permission at the root level (and is inherited within the > process group). They can start/stop processors, but cannot empty (or even > list) the queues in said process group. > > I also set up a queue at the root level and confirmed the same behaviour > there. > > Thanks > > On Thu, 4 Jun 2020 at 23:22, Bryan Bende wrote: > >> Would also add that if you don't have specific component policies on >> processors, it should inherit from the process group. So at the process >> group level you can give some users write to the actual process group which >> should control creating/deleting connections, and give some users only >> modify the data on the process group which would control clearing queues. >> >> On Thu, Jun 4, 2020 at 8:55 AM Mark Bean wrote: >> >>> Phil, >>> >>> There is a 'modify the data' Component Access Policy. Use the key icon in >>> the Operate palette (or right-click on the component) to access the >>> Component Access Policies as opposed to using the Global Menu in the >> upper >>> right to access Global Access Policies. >>> >>> The user will be able to empty a queue if they are in the 'modify the >> data' >>> policy for the upstream component (processor) which generated the data. >>> This policy does not allow the user to delete the connection between >>> processors. To do so requires the 'modify the component' policy. >>> >>> One additional nuance to consider: if you are operating a NiFi Cluster, >> you >>> will need to add each of the cluster nodes to the 'modify the data' >> policy >>> as well. This is required because the request to empty a queue is proxied >>> from the node being used to access the UI out to the remaining nodes. >>> >>> -Mark >>> >>> >>> On Thu, Jun 4, 2020 at 6:52 AM Phil H wrote: >>> >>>> Hi Andy, >>>> >>>> Thanks for your reply. I don’t recall seeing the modify data policy in >>> the >>>> user interface. Is it possible this is something I would have to change >>> at >>>> the back end? >>>> >>>> I don’t have the system in front of me now, will have to confirm >>> tomorrow. >>>> >>>> Regards, >>>> Phil >>>> >>>> On Thu, 4 Jun 2020 at 11:18, Andy LoPresto >> wrote: >>>> >>>>> Hi Phil, >>>>> >>>>> You might have uncovered a gap in the permission policy. Have you >> tried >>>>> using the “modify the data” permission [1]? If a user does not have >>> write >>>>> permission to the queue, I think they can empty it but not >>> modify/delete >>>>> the queue itself. >>>>> >>>>> I am speculating here because I haven’t had a chance to verify, but I >>>>> suspect that the same write permission which allows a user to clear >> the >>>>> queue would allow them to delete it as well. This may be something we >>>> could >>>>> mitigate by using the “operate” permission, but I would have to >>> validate >>>>> this behavior first. >>>>> >>>>> Hope this helps for now. >>>>> >>>>> [1] >>>>> >>>> >>> >> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#component-level-access-policies >>>>> >>>>> Andy LoPresto >>>>> alopre...@apache.org >>>>> alopresto.apa...@gmail.com >>>>> He/Him >>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>>> >>>>>> On Jun 3, 2020, at 4:08 PM, Phil H wrote: >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I am trying to stratify my userbase. I need to allow certain >>>> users/groups >>>>>> the ability to clear queues, but cannot find the right policy to >>> allow >>>>> that >>>>>> without also allowing them to delete queues, which I absolutely >> don’t >>>>> want >>>>>> to do. >>>>>> >>>>>> Am currently using 1.9.2 (putting off the upgrade process!) >>>>>> >>>>>> Regards, >>>>>> Phil >>>>> >>>>> >>>> >>> >>
Re: Policy to clear queues
Hi Phil, You might have uncovered a gap in the permission policy. Have you tried using the “modify the data” permission [1]? If a user does not have write permission to the queue, I think they can empty it but not modify/delete the queue itself. I am speculating here because I haven’t had a chance to verify, but I suspect that the same write permission which allows a user to clear the queue would allow them to delete it as well. This may be something we could mitigate by using the “operate” permission, but I would have to validate this behavior first. Hope this helps for now. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#component-level-access-policies Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 3, 2020, at 4:08 PM, Phil H wrote: > > Hi there, > > I am trying to stratify my userbase. I need to allow certain users/groups > the ability to clear queues, but cannot find the right policy to allow that > without also allowing them to delete queues, which I absolutely don’t want > to do. > > Am currently using 1.9.2 (putting off the upgrade process!) > > Regards, > Phil
Re: Jira contributor access request for user: "John Michaels"
Hi John, Welcome to the project. I’ve added you as a contributor to Jira. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 1, 2020, at 5:41 AM, John Michaels wrote: > > Hi All, > > I'd like to try contributing to the project, specifically with this issue > <https://issues.apache.org/jira/browse/NIFI-4356> initially. Could someone > grant me contributor access so I can assign the ticket to myself? > > Jira Username: "John Michaels" > > Thanks! > John
Re: [ANNOUNCE] New NiFi PMC member Drew Lim
Congratulations and welcome Drew. The work you’ve done makes the community so much more accessible and drives user success everywhere. I refer to your work myself all the time. Looking forward for your continued contributions. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 29, 2020, at 1:18 PM, Pierre Villard > wrote: > > Awesome Drew! Congrats! Your work is super useful for all the community, > well deserved! > > Le ven. 29 mai 2020 à 21:55, Tony Kurc a écrit : > >> NiFi Community, >> >> On behalf of the Apache NiFi PMC, I am pleased to announce that Drew Lim >> has accepted the PMC's invitation to join the Apache NiFi PMC. >> >> Drew has been doing an amazing job for several years making NiFi more >> approachable and usable. He's contributed to almost every part of NiFi; >> going over and beyond to improve documentation and usability. >> >> Please join us in congratulating and welcoming Drew to the Apache NiFi PMC. >> >> Congratulations Drew! >> >> Tony >>
Re: [ANNOUNCE] New NiFi PMC member Arpad Boda
Congratulations Arpad, very well deserved and excited to see your continued contributions. Pálinka on me next time I’m in Budapest. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 29, 2020, at 1:16 PM, Pierre Villard > wrote: > > Congrats Arpad, very well deserved! > > Le ven. 29 mai 2020 à 21:48, Tony Kurc a écrit : > >> NiFi Community, >> >> On behalf of the Apache NiFi PMC, I am pleased to announce that Arpad Boda >> has accepted the PMC's invitation to join the Apache NiFi PMC. >> >> In addition to being a regular code contributor and engaged community >> member with the project, (especially MiNiFi C++!) for quite some time, >> Arpad has done the arduous task of being a release manager for both NiFi >> Registry and MiNiFi C++, which is a challenging and detail oriented task. >> >> Please join us in congratulating and welcoming Arpad to the Apache NiFi >> PMC. >> >> Congratulations Arpad! >> >> Tony >>
Re: Nifi context path
In addition to reading the link Utkarsh provided, it’s important to note that modifying those properties informs NiFi of the proxy instance it is behind. It does not change the NiFi context paths or define the hostname, port, and path that NiFi will be accessed via. The solution to this requirement is to deploy a reverse proxy (nginx, HAProxy, etc.) in front of NiFi and redirect traffic from http://localhost:8080 <http://localhost:8080/> -> http://localhost:8080/nifi <http://localhost:8080/nifi> if that is the deployed location. Koji Kawamura has an example configuration and description available here [1]. [1] https://github.com/ijokarumawak/nifi-reverseproxy <https://github.com/ijokarumawak/nifi-reverseproxy> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 27, 2020, at 12:09 AM, Peter Turcsanyi wrote: > > Hi Ganesh, > > Please subscribe to the mailing list in order to get the replies: > http://nifi.apache.org/mailing_lists.html > You can find the archived mails there too. > > Regards, > Peter > > On Wed, May 27, 2020 at 6:32 AM Ganesh, B (Nokia - IN/Bangalore) < > b.gan...@nokia.com> wrote: > >> Hi , >> >> Can anyone respond please ? >> >> Thanks & Regards, >> Ganesh.B >> >> From: Ganesh, B (Nokia - IN/Bangalore) >> Sent: Tuesday, May 26, 2020 10:18 AM >> To: 'dev@nifi.apache.org' >> Subject: Nifi context path >> >> Hi , >> >> I am accessing the nifi UI with http://localhost:8080/nifi in my VM or >> windows machine . >> >> I want to configure Nifi to access with our own context path i.e >> http://localhost:8080/ >> >> I tried configuring the nifi properties like below . >> nifi.web.proxy.context.path="/ " >> nifi.web.proxy.host=localhost:8080 >> >> For one of the requirement we need to change the config path form "/nifi" >> to "/" to access nifi with " http://localhost:8080/ " could anyone help >> me to do the same or the possibility to do . >> >> Thanks & Regards, >> Ganesh.B >> >> >>
Re: how to externalize the properties of various processors for a given flow
You should not use bootstrap.conf for this. That file is for application-level configuration values to start the NiFi framework. I would suggest learning about the flow development lifecycle [1] using NiFi Registry [2], a complementary application which was introduced for just this purpose. This allows for versioning flow snippets, promoting them between environments, and deploying them to new NiFi instances. There are also tools like NiPyAPI [3] and the NiFi CLI Toolkit [4] which allow for scripted interaction with the flow, and the NiFi REST API [5] which allows population of arbitrary processor properties from any client. [1] https://www.slideshare.net/Hadoop_Summit/sdlc-with-apache-nifi <https://www.slideshare.net/Hadoop_Summit/sdlc-with-apache-nifi> [2] https://nifi.apache.org/registry <https://nifi.apache.org/registry> [3] https://nipyapi.readthedocs.io/en/latest/ <https://nipyapi.readthedocs.io/en/latest/> [4] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI [5] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 27, 2020, at 5:19 AM, krishna wrote: > > Hi Nifi Team, > > I am new to NIFI and learning about it. I am developer with experience in web > development. > > Do we have a way to externalize the properties that we set for each > processor. At present this is done when we define the processors in our flow > from the Web UI of NIFI. This is ok for tyring out with the processors. > > But what if we have designed our flow and moving to other environments. How > can we set the properties of each processor so that nifi can read them from > external properties file or a database. So that once we make sure we have > right properties configured, we can just concentrate on deployment of the > flows and testing. Do we have any feature to do this, or any processor that > can read external file and set all the properties of the processors in the > flow ahead. > > I found a stackover flow article to use bootstrap.config for the time being. > Are there any features added in this regard. > > -- > > Regards, > > Krishna. >
Re: Counting the number of occurrences of a phrase
David,
This would be a pretty simple Groovy script to write (maybe 2-3 lines). Have
you tried ExecuteScript?
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On May 25, 2020, at 3:51 AM, DAVID SMITH
> wrote:
>
> Good morning devs
> I have a flowfile which contains a space separated log file, within this file
> is a section such as:-
> FileSelectors:[{fileselector:},{fileselector:},{fileselector:}]
> I would like to create an attribute which is the count the number of
> occurrences of the word fileselector and ideally another attribute which is
> the filenames in one list, although this one is just a nice to have. I have
> made the whole phrase an attribute and then tried various things in the
> expression language but nothing seems to work.Can anyone help please ?
> Many thanks Dave
Re: [DISCUSS] Bringing MiNiFi into NiFi
Jeff, my understanding is that MiNiFi C++ development will continue unaffected by this. While the release process may shift slightly as MiNiFi Java and NiFi are combined, I actually think this will improve both processes and shouldn’t negatively influence the capability to release either in a useable format. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 21, 2020, at 12:22 PM, Jeff Zemerick wrote: > > Apologies if this has been discussed previously. I've not kept up on the > Mi/NiFi progress as much lately. > > When I think of two projects being combined I usually consider the > connection between the lifecycles of the two projects. One thing I always > liked about MiNiFi being separate was its ability to evolve outside the > NiFi lifecycle. New features, enhancements, and fixes could be released as > needed. It also had its own voting process for releases. But it certainly > had its drawbacks, too. > > With the two projects being combined, is there any fear of negative effects > to MiNiFi's development being tied to NiFi's release cadence? Will it be > possible to do a MiNiFi release outside of a NiFi release? Or any desire to? > > I'm not at all proposing not to merge the projects because there's a lot to > gain as stated in the initial email, but I would like to see MiNiFi be able > to maintain some of that independence and flexibility, if possible. > > Thanks, > Jeff > > On Thu, May 21, 2020 at 1:54 PM Joe Witt wrote: > >> Yes please! >> >> On Thu, May 21, 2020 at 1:53 PM Andy LoPresto >> wrote: >> >>> Matt, >>> >>> I think this is a great idea, and I think it could also provide a bridge >>> to the reduced kernel build of NiFi with separate extensions that the >>> extensions registry will ultimately offer. Once the MiNiFi and NiFi >>> assemblies are complete, it should be possible to add a third which does >>> include the UI/API, but does not include the majority of specialized >>> processor NARs, etc. >>> >>> Andy LoPresto >>> alopre...@apache.org >>> alopresto.apa...@gmail.com >>> He/Him >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>> >>>> On May 21, 2020, at 10:49 AM, Pierre Villard < >>> pierre.villard...@gmail.com> wrote: >>>> >>>> Nothing useful to add other than I know this is going to be a lot of >>> work, >>>> but this is GREAT! >>>> >>>> Thanks, >>>> Pierre >>>> >>>> Le jeu. 21 mai 2020 à 19:44, Matt Burgess a >>> écrit : >>>> >>>>> All, >>>>> >>>>> I'm currently working on MINIFI-422, which aims to bring the MiNiFi >>>>> code into the NiFi codebase and have the MiNiFi product be a >>>>> specialized assembly of NiFi. Picture two different Maven profiles >>>>> that create a NiFi assembly or a MiNiFi assembly, each including the >>>>> common elements but excluding those things that aren't needed, such as >>>>> MiNiFi being "headless" and not including Jetty or the UI. >>>>> >>>>> This will be a lot of effort and very invasive to NiFi itself, so I >>>>> created a MINIFI-422 branch (based on the current master) and pushed >>>>> that to the Apache NiFi Github repo. I figure that way we can carve >>>>> out individual tasks and write PRs against that branch, rather than >>>>> duplicating code in the meantime and having MiNiFi show up little by >>>>> little in the NiFi codebase. >>>>> >>>>> My first task to that end is to continue Aldrin's work on MINIFI-488. >>>>> I originally had a PR against master for this subtask, but after >>>>> discussing with Aldrin we thought it would be better to have a >>>>> separate branch and incrementally add MiNiFi functionality until we're >>>>> ready for a big PR to bring it all into NiFi. >>>>> >>>>> We still want to keep up with NiFi master, so the branch would be >>>>> subject to force-pushes, rebases, etc. For that reason I'd like to ask >>>>> that anyone working on that branch please reach out to me >>>>> (mattyb...@apache.org) so we can coordinate and collaborate, to >> ensure >>>>> we're not stepping on each other's toes :) >>>>> >>>>> Also happy to discuss alternate approaches or any other questions or >>>>> concerns you may have. >>>>> >>>>> Regards, >>>>> Matt >>>>> >>> >>> >>
Re: [DISCUSS] Bringing MiNiFi into NiFi
Matt, I think this is a great idea, and I think it could also provide a bridge to the reduced kernel build of NiFi with separate extensions that the extensions registry will ultimately offer. Once the MiNiFi and NiFi assemblies are complete, it should be possible to add a third which does include the UI/API, but does not include the majority of specialized processor NARs, etc. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 21, 2020, at 10:49 AM, Pierre Villard > wrote: > > Nothing useful to add other than I know this is going to be a lot of work, > but this is GREAT! > > Thanks, > Pierre > > Le jeu. 21 mai 2020 à 19:44, Matt Burgess a écrit : > >> All, >> >> I'm currently working on MINIFI-422, which aims to bring the MiNiFi >> code into the NiFi codebase and have the MiNiFi product be a >> specialized assembly of NiFi. Picture two different Maven profiles >> that create a NiFi assembly or a MiNiFi assembly, each including the >> common elements but excluding those things that aren't needed, such as >> MiNiFi being "headless" and not including Jetty or the UI. >> >> This will be a lot of effort and very invasive to NiFi itself, so I >> created a MINIFI-422 branch (based on the current master) and pushed >> that to the Apache NiFi Github repo. I figure that way we can carve >> out individual tasks and write PRs against that branch, rather than >> duplicating code in the meantime and having MiNiFi show up little by >> little in the NiFi codebase. >> >> My first task to that end is to continue Aldrin's work on MINIFI-488. >> I originally had a PR against master for this subtask, but after >> discussing with Aldrin we thought it would be better to have a >> separate branch and incrementally add MiNiFi functionality until we're >> ready for a big PR to bring it all into NiFi. >> >> We still want to keep up with NiFi master, so the branch would be >> subject to force-pushes, rebases, etc. For that reason I'd like to ask >> that anyone working on that branch please reach out to me >> (mattyb...@apache.org) so we can coordinate and collaborate, to ensure >> we're not stepping on each other's toes :) >> >> Also happy to discuss alternate approaches or any other questions or >> concerns you may have. >> >> Regards, >> Matt >>
Re: Jira contributor access
Hi Stefan, Are you requesting contributor access to the Apache NiFi Jira instance, and if so, what is your username? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 18, 2020, at 10:38 AM, Stefan Kok wrote: > > kok
Re: [discuss] 1.12.0 or 1.11.5...
Thanks for starting this thread Joe. I agree the back-pressure fix is one that will help the community a lot. I am currently working on a couple security features which would also greatly benefit users. I’d like to see these get in 1.12.0 if possible, especially as there are some changes that have already gone into master which I don’t think are ready to be released and then required to be supported. Some are already up as PRs. My preferences, ranked: 1. Wait a few days on 1.12.0, release it with the new security features and the back-pressure fix (along with the other 170+) 2. Release 1.11.5 immediately for the back-pressure fix 3. Release 1.12.0 immediately for the back-pressure fix and the security features have to wait for 1.13.0 Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 13, 2020, at 10:52 AM, Joe Witt wrote: > > Team, > > A couple folks have recently reported slowness in the UI in the recent > releases particularly with larger and more production style flows. It > appears to be related to when back pressure prediction is enabled [1]. > > It also appears this is already fixed [2] and on 1.12.0-SNAPSHOT. > > I'd like to produce a release which captures this. It isn't clear that > 1.12 is ready to roll but happy to look into that path if so. There are > already 177+ closed JIRAs on that thing!! > > Alternatively if easier I could kick out another on 1.11 which would be > 1.11.5 and go looking opportunistically for other bits to grab as well. > > Any thoughts? > > [1] https://issues.apache.org/jira/browse/NIFI-7437 > [2] https://issues.apache.org/jira/browse/NIFI-7087 > > Thanks > Joe
Re: Question about LDAP authentication/authorization
Hi Claudio, Yes, you should be able to accomplish this by putting the highest level group you want authorized (in this case, Group B) as the target. There is more information available in the Admin Guide [1]. I think the important setting for this case is Group Search Scope, which would likely need to be set to “SUBTREE”. More information on nested LDAP groups here [2]. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider> [2] https://stackoverflow.com/a/5135252/70465 <https://stackoverflow.com/a/5135252/70465> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On May 7, 2020, at 11:18 AM, Lombardo, Claudio > wrote: > > Hello there. > > I was wondering if NIFI supports nested groups for authorization. > > Basically, if a user is a member of Group A and group A is a member of Group > B, is it possible to have NIFI authorize group B, which only contains another > group (group A)? > > Thank you, > Claudio > > Claudio M. Lombardo > Sr. Enterprise Platforms Engineer > Application Platforms > Information Technology Services > > UC San Diego > cmlomba...@ucsd.edu
Re: CSVRecordSetWriter Flow File Content is Empty
Hi James, Mark’s approach is right. Again, without seeing your code, I would throw out that a common issue in custom processor development is performing the session.write(), session.putAttribute(), etc. operations but failing to _assign the result back to the flowfile variable_. The variable you pass to the output relationships has to have the “most current” version of the flowfile to result in the modified content being successfully transferred. If you’re able to share the processor code, we can probably help diagnose further. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 29, 2020, at 9:44 AM, Mark Payne wrote: > > James, > > It’s hard to say without having your code, but generally if you’re seeing the > output not containing any information, it means that the schema doesn’t match > the Record given to it. The best approach is probably to just set a > breakpoint in the CSVRecordSetWriter so that you can see the incoming Record > object and see which fields it’s attempting to pull from the Record to > understand why it’s not writing anything out. > > Thanks > -Mark > > >> On Apr 29, 2020, at 12:29 PM, james medel wrote: >> >> Hi I have been working on a custom processor "ExecuteMojoScoringRecord" that >> integrates H2O MOJO Scoring Pipeline with NiFi to do batch scoring and real >> time scoring on test data to get predicted label(s). In an earlier test of >> the processor, I used a hydraulic system sensor data set and only predicted >> 1 label, which was cooling efficiency. The flow file outputted by the >> "ExecuteMojoScoringRecord" had content of 1 predicted label field name and >> field value. >> >> However, later when I updated the Hydraulic data set by adding a cooling >> condition label as the last column, now the processor's MOJO was predicting >> cooling condition, which returns 3 predicted labels ""cool_cond_y_3, >> cool_cond_y_20, cool_cond_y_100" and their associated prediction values. >> However, the flow files coming out of this processor have empty content. >> The prediction values are missing. Right before the >> writer.write(scoredFirstRecord), I added logs into the code to see if the >> key value pairs of the scoredFirstRecord contained the field names and field >> values. The data showed up in the logs. Yet, it does not show up the flow >> file content. The changes I made to the output Avro schema were adding 3 >> field names and their field data types, which are double cause the >> prediction data type is Float64. Why do you think the flow file content is >> empty? >> >
Re: Record Readers and Writers
Hi Dave, The underlying internal “record format” is not JSON. Avro [1] is used to describe schemas across all record formats, but the internal data storage is NiFi specific. You may be interested in these articles by Mark Payne and Bryan Bende [2][3][4] and the potential use of the ScriptedReader [5] or ScriptedRecordSetWriter [6] to prototype your needed conversions. [1] https://avro.apache.org/ <https://avro.apache.org/> [2] https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi <https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi> [3] https://blogs.apache.org/nifi/entry/real-time-sql-on-event <https://blogs.apache.org/nifi/entry/real-time-sql-on-event> [4] https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries <https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries> [5] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.11.4/org.apache.nifi.record.script.ScriptedReader/index.html [6] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.11.4/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 21, 2020, at 6:01 AM, DAVID SMITH > wrote: > > Hi > I want to use the ConvertRecord Processor with it's underlying Record Readers > and Writers to convert files from XML or JSON to a bespoke format and > probably vice versa.I have looked at the Readers/Writers currently provided > and decided that I can use the XML/JSON ones provided but I will need to > write something for the bespoke format. So I started looking at the current > source code for the Readers/Writers to see how they work and what I would > need to do. When running the unit tests on the XMLReader I notice on the > console that the output is in JSON format.My question is, is JSON the common > format that all records are converted to and from? > Also is there any specific documentation on writing Reader/Writers, I have > only found the developers guide? > Many thanksDave >
Re: Persisting/loading NIFI flows for Docker deployment
Thanks for that context and perspective, Kevin. Good luck on your project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 13, 2020, at 6:07 AM, Kevin Telford wrote: > > Hi all - thank you for all the thoughtful feedback. > > Regarding my original question, I think the patterns Mike outlined would be > good enough. > That said, we're not going to move forward using NiFi for the project, and I > figured I'd take a step back to explain where we were coming from, as some > may find the perspective useful. Or not :) > > > We have a project that needs some data transformation. Input is excel, output > multiple CSVs or POSTs of data to an API. On the surface, simple enough. > > Our input Excel can and will change a lot, so we'll need rapid iterations, > and testing. > > The project architecture is container-based, currently consisting of a front > end docker image, a back end image, and database image. ETL is intended to be > a fourth. It can be orchestrated with Docker Compose, K8, or bare metal. The > goal is to be turn key and low friction. > > There were two reasons we didn't choose NiFi - the painful (read: long) Java > deployment lifecycle for custom processing, and system complexity, > particularly around updating new flows. > > Regarding the pain of Java, I've partied with Java since 1.4, so I get it. > But these days, if I have a data analyst/data engineer with lowish > programming skills, I can't have them compiling and moving around jars, nor > do I want to invest in building out the build/deploy pipeline. Platforms have > really evolved (especially look at the cloud native tools), and code can be > written "in line" in the UI, and just deployed. A lot of this is due to > dynamic languages (e.g. Python), but it can still be done with Java with > behind the scenes compilation. Juypter Notebook, for it's many, many faults, > is the way things are heading, and the kids love it. > > I touched a lot on updating flows above, but in NiFi my choices seemed to be > to replace the Flow.xml.gz file, or use the NiFi Registry. My concern with > the registry was that it was yet another moving part, and even still I'd have > to build in source control workflows. Here again, newer platforms have all > this baked in. > > > In closing, I think there is definitely still a place for NiFi, especially on > the enterprise side where stability, scale and management are paramount. But > I did want to share this, as these non-enterprise use cases I am describing > will, over time become the enterprise use cases, and the NiFi project would > do well to evaluate their long term strategy. > > Thanks again for all the responses. > Best, > Kevin > > On 2020/04/08 14:27:54, Kevin Telford wrote: >> Hi all – I have a two part question. >> >> >> >> I’d like to run NiFi inside a container in order to deploy to various >> environments. As far as I can tell, the flow.xml.gz file is the main >> “source” if you will, for a NiFi data flow. >> >> Q1) Is the flow.xml.gz file the “source” of a NiFi data flow, and if so, is >> it best practice to copy it to a new env in order to “deploy” a prebuilt >> flow? Or how best is this handled? >> >> >> >> Given that Q1 is true, my challenge then becomes somewhat Docker-specific… >> >> Situation: >> >> - In the Dockerfile we unzip the NiFi source (L62 >> >> <https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L62>) >> and then create Docker volumes (L75 >> >> <https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L75> >> specifically for the conf dir). Once the container starts all the normal >> NiFi startup things happen, and /opt/nifi/nifi-current/conf/flow.xml.gz >> created. >> >> Complication: >> >> - In order to persist flow.xml.gz outside of the container, I would >> normally mount the /opt/nifi/nifi-current/conf directory, however in this >> case I cannot mount it on initialization because that will overwrite conf >> config files with whatever directory I bind it to (Docker container >> isolation ensures host -> container file precedence). >> - I could mount to a running container, but this is less ideal due to >> the various ways a container can be deployed. >> - I could copy manually from the running container, but this is less >> ideal as it’s on demand, and not always persisting latest. >> >> Resolution: >> >> - I be
Re: [Discuss]NiFi-registry 0.6.0 tagging and branching
I agree with Bryan’s plan. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 6, 2020, at 7:12 AM, Bryan Bende wrote: > > That makes sense to me. > > I think going forward we should consider switching things around a little > bit and get master back to the state of 0.6.0 and set it to 0.7.0-SNAPSHOT, > and then keep the 1.0.0 work in another branch, mainly because it is > unclear when we would be able to release a 1.0.0 registry which likely > requires a 2.0.0 NiFi. > > We can figure this out after finalizing the 0.6.0 release, so for now what > you suggested seems like the correct thing to do. > > On Mon, Apr 6, 2020 at 9:52 AM Arpad Boda wrote: > >> Hi, >> >> As you may know, 0.6.0 was technically a patch release, branching from >> master based on 0.5.0 and cherry-picking non-breaking changes from master. >> >> Because of this merging back the release branch to master would just cause >> conflict and master has the proper versions (1.0.0) already set. >> >> To avoid this issue I would just add the release tag to 0.6.0 branch and >> avoid merging. >> >> Please share your opinion, especially in case you have concerns with the >> above. >> >> Cheers, >> Arpad >>
Re: Jira contributor access
Hi Danil, Welcome to the project. I have added you as a contributor. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 3, 2020, at 10:10 PM, Danil wrote: > > Hi team, > > I'd like to contribute to the NiFi project. Could I get a contributor > access please? > My username is shpilevoy. > > Thank you! > > -- > Best regards, > Danil Shpilevoy
Re: Submitted April 2020 board report for Apache NiFi
This is really detailed and insightful Joe. Thanks for preparing it. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Apr 3, 2020, at 8:51 AM, Joe Witt wrote: > > Team, > > Here is the report I've submitted for this months ASF board meeting > > ## Description: > The mission of NiFi is the creation and maintenance of software related to > providing an easy to use, powerful, and reliable system to process and > distribute data. > > Apache NiFi MiNiFi is an edge data collection agent built to seamlessly > integrate with and leverage the command and control of NiFi. There are both > Java and C++ implementations. > > Apache NiFi Registry is a centralized registry for key configuration items > including flow versions, assets, and extensions for Apache NiFi and Apache > MiNiFi. > > Apache NiFi Nar Maven Plugin is a release artifact used for supporting the > NiFi classloader isolation model. > > Apache NiFi Flow Design System is a theme-able set of high quality UI > components and utilities for use across the various Apache NiFi web > applications in order to provide a more consistent user experience. > ## Issues: > There are no issues requiring board attention at this time. > > ## Membership Data: > Apache NiFi was founded 2015-07-14 (4 years ago) > There are currently 47 committers and 30 PMC members in this project. > The Committer-to-PMC ratio is roughly 3:2. > > Community changes, past quarter: > - No new PMC members. Last addition was Peter Wicks on 2019-05-29. > - No new committers. Last addition was Peter Turcsanyi on 2019-10-25. > > We have strong committer and PMC pipeline to consider so we expect to see > activity here pending discussions by the PMC. > > ## Project Activity: > Apache NiFi Registry 0.6.0 is currently under Release Candidate vote. It is > mostly stability and security related changes. > > Apache NiFi 1.11.0 through 1.11.4 have all been released in January, > February, and March respectively. The 1.11 release line brought in some > awesome features like better integration with Azure, all NiFi repositories > can now be encrypted at an application level, class loader isolation now > works across nars with native libraries, and more. We've also addressed > a massive number of bugs, improvements, and security related fixes. > > Apache NiFi MiNiFi CPP 0.7.0 was released in January with 145 issues > addressed. > It includes a number of new features like SFTP support, running as a > windows > service, tail support for globs/wildcards, windows event log consumption, > and > a long list of stability improvements. > ## Community Health: > In our previous reporting period we noted a decline of roughly 15% mailing > list activity in dev and users and attributed that to the time of year and > the > rise of engagements in slack. Slack engagement continues to rise but this > period we see an increase of 58% on our dev list and 16% on users and also a > large increase in issues activity of 61%. The community is very busy > including more than a 100% increase in commits and 17% increase in the > number > of contributors of committed code in the quarter. Anecdotally a significant > increase in new user activity is observed as well. Many mailing list > questions are starting from very limited knowledge and folks are looking > for a > lot of assistance with patterns and getting started in terms of pure usage. > > We see also constant activity for Apache NiFi on Twitter, Stackoverflow, > Youtube, etc. These are for tutorials, example use cases, job/ads, questions > and answer created by people active in the community and not. > > For the past few quarters we've reported 394, 523, 707, and now 895 > persistent > users in our slack channels. While it appeared this might harm or reduce > mailing list activity that doesn't seem to be the case. We've just opened > up > more ways for folks to collaborate in the community. The slack channels are > extremely busy and the depth of questions range from very superficial > questions easily answered to deeper and more complex situations that lead to > JIRAs and mailing list discussions. Mostly though it just appears to be a > communication mode some users and developers like far more than the lists. > > Our community activity level has certainly increased while at the same time > we > have not increased committer or PMC ranks. This is really a reflection of > the > fact that the PMC has been very focused on knocking out releases and > engaging > on the various mailing list and slack channels. We need to remind ourselves > to groom and manage the committer/PMC pipeline and there are least a couple > candidates well positioned for committer and/or PMC status based on past > discussions. Overall the state of the community appears very strong.
Re: Questions about NiFi source code build
If you just build the module you’re working on but don’t recompile any of the dependent follow-on modules, or put the built NAR into the lib/ directory of the running NiFi instance and restart (depending on what you’ve changed), you will get errors similar to what you’re referencing. The safest & easiest way is to rebuild the whole project from the root. The faster but more complicated way is to determine what depends on what you’ve changed and rebuild and deploy things surgically. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 23, 2020, at 12:37 PM, Waleed Aibany wrote: > > Thank you all, > > The build working fine. > > I added a new code lines, then I just built module that I am working on by > this command: [ mvn -T C2.0 clean install -Ddir-only], when I try to debug > it to see my chnages, it shows me this on Breakpoint: > ([image: skip.PNG] no executable code found at line). My changes not > reflects on nifi canvas. > > Thank you again :) > > في الاثنين، 23 مارس 2020 في 4:31 م تمت كتابة ما يلي بواسطة Mike Thomsen > <mikerthom...@gmail.com>: > >> Also, I think even in a multi-module build like NiFi just running "package" >> will not make compiled dependencies available. Ergo the JAR file for the >> Mongo client service won't be available to the NiFi modules that reference >> it unless you switch to the install goal. >> >> On Sun, Mar 22, 2020 at 6:08 PM Joe Witt wrote: >> >>> Muazma’s reply was spot on. An additional point is to do the build from >>> the root nifi source directory. >>> >>> On Sun, Mar 22, 2020 at 5:41 PM Muazma Zahid wrote: >>> >>>> mvn -Pcontrib-check clean install //for contribution check >>>> mvn clean install –DskipTests //for skipping tests >>>> >>>> //works on MAC OS >>>> >>>> export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m" >>>> >>>> mvn -T C2.0 clean install >>>> More information at http://nifi.apache.org/quickstart.html >>>> >>>> Thanks, >>>> Muazma >>>> >>>> On Sun, Mar 22, 2020 at 2:20 PM Waleed Aibany >>> wrote: >>>> >>>>> Dear Nifi Team, >>>>> >>>>> What is the maven command you use to build NiFi Source Code? Any >>>> preferable >>>>> one!. I face some issues when I try to run [mvn package], it is give >> me >>>>> this message as example "*The POM for >>>>> >> org.apache.nifi:nifi-mongodb-client-service-api-nar:nar:1.12.0-SNAPSHOT >>>> is >>>>> missing, no dependency information available*". >>>>> >>>>> Thank you >>>>> >>>> >>> >>
Re: Nifi Enhancement Idea
Hi Rifat, The prefix property is to filter by the hierarchical structure S3 exposes [1]. I don’t believe there is a native way to use string comparisons or regular expressions on the remainder of the object keys list in S3. Rather than have a “suffix” property which only allows object-level filtering on one end of the name, I think a “pattern” property which allows regular expression matching would be useful. However, this would need to use the Java SDK [2] and operate locally on S3ObjectSummary objects which have been retrieved from the server. So this could certainly reduce the number of flowfiles output from the processor (i.e. only objects which end in .csv would be created as flowfiles) but the whole list (e.g. .csv + .pdf + .txt) would be retrieved from the S3 server and filtered within the ListS3 processor. You can submit a Jira ticket for a feature request here [3]. [1] https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html <https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html> [2] https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingJava.html <https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingJava.html> [3] https://issues.apache.org/jira/secure/CreateIssue!default.jspa <https://issues.apache.org/jira/secure/CreateIssue!default.jspa> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Mar 18, 2020, at 11:49 AM, Evans, Jason D. > wrote: > > Hey Rifat – cool idea. Is the ListS3 processor something we wrote? Or are > you suggesting contributing to Nifi itself? > > From: "Chowdhury, Rifat" > Date: Wednesday, March 18, 2020 at 2:45 PM > To: "dev@nifi.apache.org" > Cc: "Loyot, Brendan" , "Nagwani, Dheeraj X. -ND" > , "Evans, Jason D." > , "Rudra, Anshuman X. -ND" > > Subject: Nifi Enhancement Idea > > Hi, > > For processors such as ListS3, would it be beneficial to add a new field > called “suffix” so that when I configure my ListS3 processor, I can filter my > incoming files furthers in addition to prefix? The current Template I follow > is ListS3 -> RouteOnAttribute(filename endsWith my suffix criteria) Like if I > want to filter for only .csv or .tsv or .gz or .txt files or any other types > of files. Or another Option is have prefix field accept some sort of Java > REGEX Pattern matching. What do you guys think? Shall I create a PR for this > against Nifi Branch for future releases? I don’t feel the need to create a > custom processor for this rather small enhancement. > > Best Regards, Rifat Chowdhury > Software Engineer, Data Platforms > > THE WALT DISNEY COMPANY | DIRECT-TO-CONSUMER AND INTERNATIONAL > rifat.chowdh...@disney.com <mailto:rifat.chowdh...@disney.com>
Re: Managing custom processor logging statements during JUnit testing
Russell,
You can put a file called logback-test.xml in the src/test/resources/ directory
of your custom processor module. This file is the same format as logback.xml
but will supersede it for unit test execution. You can configure whatever level
of logging severity and specificity in the same way you would in the production
file.
Here is an example in nifi-security-utils [1].
[1]
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/test/resources/logback-test.xml
<https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/test/resources/logback-test.xml>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 6, 2020, at 12:13 PM, Russell Bateman wrote:
>
> I'm interested in getting log statements that may occur from my *custom
> processor* code at the INFO, DEBUGand TRACElevels when running unit tests for
> it. This means that I would like to set (programmatically at runtime or by
> configuration) what I am used to setting in
> /${NIFI_ROOT}///conf/logback.xml/, but effectively in/during my unit tests
> (using NiFi's TestRunner). And, obviously, I'd like to see those log
> statements come out to the console.
>
> Thanks for any advice.
>
Re: 1.11.3 trust store error
You can post them on a temporary file sharing service, post them in the Apache NiFi Slack [1], or email them to me directly at alopre...@apache.org <mailto:alopre...@apache.org>. The mailing list software tends to strip attachments. [1] https://apachenifi.slack.com <https://apachenifi.slack.com/> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 26, 2020, at 12:21 PM, Joe Gresock wrote: > > Good question -- I can't share these keystores and truststores, but I'll > see if I can generate some test ones tomorrow. How should I send them to > you? > > On Wed, Feb 26, 2020 at 8:01 PM Andy LoPresto wrote: > >> Joe, >> >> Can you share the keystores and truststores you are using? I understand >> the issue you’re encountering but we haven’t yet been able to reproduce it >> locally running with certs that work on 1.11.1. Please DO NOT share actual >> keystores if they contain real private keys, only if these are dev >> instances you are comfortable sharing. >> >> If you cannot share these, can you please try generating a new set using >> the TLS Toolkit and verify that they also fail on 1.11.3 in your >> environment? >> >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Feb 26, 2020, at 11:07 AM, Joe Gresock wrote: >>> >>> Ok, I added all the server certs and my administrator's client cert to >> the >>> trust store, and they all still got PKIX path building failed. So I >>> redeployed nifi 1.11.1, and now it works again. >>> >>> Joe >>> >>> On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock wrote: >>> >>>> Yes, on Kubernetes. >>>> >>>> FWIW, I do see changes to >>>> >> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java >>>> involving a new function createTrustSslContextWithTrustManagers(), among >>>> other related changes. >>>> >>>> I'm going to try directly adding the client certs to my trust store to >> see >>>> if that works (instead of just adding the client certs' issuer). >>>> >>>> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt wrote: >>>> >>>>> on kubernetes is a key detail here too... >>>>> >>>>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock >> wrote: >>>>> >>>>>> Java 8 >>>>>> >>>>>> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough >>>>> wrote: >>>>>> >>>>>>> Hi Joe, >>>>>>> >>>>>>> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any >>>>>>> issues like you describe. >>>>>>> >>>>>>> Are you running Java 8 or Java 11? >>>>>>> >>>>>>> Nathan >>>>>>> >>>>>>> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock >>>>> wrote: >>>>>>> >>>>>>>> Were there any changes with how the trust store is used in 1.11.3? >>>>> I >>>>>>> had a >>>>>>>> 1.11.0 deployment working with the following settings, but when I >>>>>>> deployed >>>>>>>> 1.11.3, the cluster can't seem to replicate requests to itself: >>>>>>>> >>>>>>>> nifi.remote.input.host= >>>>>>>> nifi.remote.input.secure=true >>>>>>>> nifi.remote.input.socket.port=32440 >>>>>>>> nifi.remote.input.http.enabled=true >>>>>>>> >>>>>>>> nifi.cluster.protocol.is.secure=true >>>>>>>> nifi.cluster.is.node=true >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>>>> nifi.cluster.node.protocol.port=6007 >>>>>>>> >>>>>>>> >>>>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>>>> nifi.web.https.port=8443 >>>>>>>> >>>>>>>
Re: 1.11.3 trust store error
Joe, Can you share the keystores and truststores you are using? I understand the issue you’re encountering but we haven’t yet been able to reproduce it locally running with certs that work on 1.11.1. Please DO NOT share actual keystores if they contain real private keys, only if these are dev instances you are comfortable sharing. If you cannot share these, can you please try generating a new set using the TLS Toolkit and verify that they also fail on 1.11.3 in your environment? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 26, 2020, at 11:07 AM, Joe Gresock wrote: > > Ok, I added all the server certs and my administrator's client cert to the > trust store, and they all still got PKIX path building failed. So I > redeployed nifi 1.11.1, and now it works again. > > Joe > > On Wed, Feb 26, 2020 at 6:21 PM Joe Gresock wrote: > >> Yes, on Kubernetes. >> >> FWIW, I do see changes to >> nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/SslContextFactory.java >> involving a new function createTrustSslContextWithTrustManagers(), among >> other related changes. >> >> I'm going to try directly adding the client certs to my trust store to see >> if that works (instead of just adding the client certs' issuer). >> >> On Wed, Feb 26, 2020 at 6:08 PM Joe Witt wrote: >> >>> on kubernetes is a key detail here too... >>> >>> On Wed, Feb 26, 2020 at 10:01 AM Joe Gresock wrote: >>> >>>> Java 8 >>>> >>>> On Wed, Feb 26, 2020 at 5:59 PM Nathan Gough >>> wrote: >>>> >>>>> Hi Joe, >>>>> >>>>> I just set up a secure cluster with NiFi 1.11.3 and am not seeing any >>>>> issues like you describe. >>>>> >>>>> Are you running Java 8 or Java 11? >>>>> >>>>> Nathan >>>>> >>>>> On Wed, Feb 26, 2020 at 12:22 PM Joe Gresock >>> wrote: >>>>> >>>>>> Were there any changes with how the trust store is used in 1.11.3? >>> I >>>>> had a >>>>>> 1.11.0 deployment working with the following settings, but when I >>>>> deployed >>>>>> 1.11.3, the cluster can't seem to replicate requests to itself: >>>>>> >>>>>> nifi.remote.input.host= >>>>>> nifi.remote.input.secure=true >>>>>> nifi.remote.input.socket.port=32440 >>>>>> nifi.remote.input.http.enabled=true >>>>>> >>>>>> nifi.cluster.protocol.is.secure=true >>>>>> nifi.cluster.is.node=true >>>>>> >>>>>> >>>>> >>>> >>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>> nifi.cluster.node.protocol.port=6007 >>>>>> >>>>>> >>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local >>>>>> nifi.web.https.port=8443 >>>>>> >>>>>> nifi.security.keystore=./conf/keystore.jks >>>>>> nifi.security.keystoreType=jks >>>>>> nifi.security.keystorePasswd= >>>>>> nifi.security.keyPasswd= >>>>>> nifi.security.truststore=./conf/truststore.jks >>>>>> nifi.security.truststoreType=jks >>>>>> nifi.security.truststorePasswd= >>>>>> nifi.security.needClientAuth=true >>>>>> >>>>>> A trusted client cert that worked against the old cluster is getting >>>> the >>>>>> same trust error (PKIX path building failed). I've verified that >>> the >>>>>> client cert was issued by an issuer that is definitely in the >>>>>> ./conf/truststore.jks as a trustedCertEntry. >>>>>> >>>>>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7] >>>>>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator >>>>>> javax.net.ssl.SSLHandshakeException: >>>>>> sun.security.validator.ValidatorException: PKIX path building >>> failed: >>>>>> sun.security.provider.certpath.SunCertPathBuilderException: unable >>> to >>>>> find >>>>>> valid certification path to r >>>>>> equested target >>>>>>at sun.security.ssl.Alerts.getSS
Re: Grant contributor access
Davy, I added you. Welcome to the project. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 21, 2020, at 11:32 AM, Davy Machado wrote: > > Guys, > > Could you please add me as a contributor in Apache NiFi'ss Jira? > User: davy.machado > > Thanks! > > Regards, > Davy Alvarenga Machado
Re: [DISCUSS] Advanced search capabilities
Bence,
This is a really well-crafted proposal and I think it would be helpful to a lot
of people. Are you comfortable opening a Jira for this feature? I think in
addition to providing the functionality, we would need to generate some helpful
and clear documentation and perhaps even suggest the filters in the search
UI/UX, as people will only use the feature if they are aware of it.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 21, 2020, at 10:42 AM, Martin Ebert wrote:
>
> We still think about building a graph based search (Neo4j) in top of NiFi.
> Would be also fantastic to have it within NiFi.
>
> There are plenty of examples
> https://blog.grandstack.io/using-neo4js-full-text-search-with-graphql-e3fa484de2ea
> From the idea it could go in this direction - of course much more
> rudimentary. Then one would have the possibility to have only the results
> displayed as text or to find out exploratory connections (graph layout).
> The built-in data lineage function of NiFi would also benefit from the
> power of Neo4j.
>
> Simon Bence schrieb am Fr., 21. Feb. 2020, 19:00:
>
>> Dear Community,
>>
>> In my project, I do use relatively high number of processors and process
>> groups. The current search function on the NiFi UI has no capabilitites to
>> narrow the results based on the group, which would make the results more
>> relevant, so I would like to propose a possible solution. Please if you
>> have any comment on this, do not hesitate to share it.
>>
>> The general approach would be to keep the current text box and extend the
>> server side capabilities to process search query in the similar manner for
>> example the Google search behaves.This extensions I would call "filters".
>> For now I am interested in the ones I will mention below, but I think, it
>> is only a matter of small work for further extend the solution with further
>> ones.
>>
>> In order to distinguish the filters from the rest of the search query, I
>> propose to put them at the beginning of the query and use the
>> [a-zA-Z0-9\.]{1..n}\:[a-zA-Z0-9\.]{1..n} format. For example a filter might
>> look the following: lorem:ipsum
>>
>> Adding this, the search query should look like the following:
>>
>> filter1:value filter2:value rest of the query
>>
>> As for processing the filters, I suggest the following behaviour:
>>
>> - Without filters the current behaviour should be kept
>> - Everything after the filters should be handled as the search term
>> - After the first "non filter word", anything should be considered as part
>> of the search term (meaning: to keep the text parsing simple, I would not
>> go in the direction to support filters at the end of the query, etc.)
>> - The ordering of the filters should have no effect on the result
>> - Filter duplications should be eliminated
>> - In case a filter appears multiple times in the query, the first occasion
>> will be used
>> - Unknown filters should be ignored
>> - Only adding filters will not end up with result, at least one character
>> must appear as search term
>>
>> Suggested filters:
>>
>> scope
>> Narrows the search based on the user's currently active process group. The
>> allowed values are: "all" and "here". All produces the current behaviour,
>> thus no filtering happens, but "here" should use the current process group
>> as "root" of the search, ignoring everything else (including parent group).
>> Note: This needs a minimal frontend change, because as I did see, currently
>> the current group is not sent with the search query.
>>
>> group
>> Narrows the search for a given processing group, if it exists. The
>> behaviour is recursive, thus the result will include the contained groups
>> as well. If it is a non-existing group, the result list should be empty.
>>
>> properties
>> Controls if properties values are included or not. If not provided, the
>> property values will be included. This is because in a lot of cases there
>> is a huge number of results come from property names.
>>
>> - Valid values for inclusion: yes, true, include, 1
>> - Valid values for exclusion: no, none, false, exclude, 0
>>
>> It is possible that the range of possible values should be limited (and not
>> being ambiguous), but I see a merit of "permissiveness" here as it is
>> simpler to remember.
>>
>> Also some example:
>>
>> 1.
>> scope:here proper
Re: Pull from NiFi Registry on startup in a new instance of NiFi
Jon, I think your idea is interesting but there are a lot of moving parts here with negotiating a connection, validating security, and defining how the communication between NiFi and Registry would work. Those definitions are not stored in the nifi,properties file, and introducing them there would be difficult. There is, however, a simpler solution to your request. You can populate a flow.xml.gz in your NiFi container so that it has that flow definition without needing to join an existing cluster. You can make that flow definition static or dynamic by using the NiFi CLI Toolkit to retrieve a specific flow definition from the registry instance as part of your container setup process. This should meet the requirements you’ve listed below. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 20, 2020, at 12:00 PM, Kessler, Jon wrote: > > I'm interested in being able to configure NiFi in a way that it will, upon > startup, create a process group using the latest (or otherwise specified) > flow that is stored within a NiFi Registry (probably something in > nifi.properties). If it fails to connect to/pull from the registry then I > want that NiFi to fail to start. I think this would be useful when it comes > to deploying NiFi in a container so that you can have it start with a > specific flow without having to have it join an active NiFi cluster. That > process group (or groups if we want to get fancy and allow more than one) > should be configured so that version control with the registry is already set > up as well. > > This behavior should only occur if the NiFi being started currently has an > empty flow. This should prevent changes to the graph from being stomped on if > that NiFi is restarted later on. > > Just wanted to solicit thoughts on such a thing.
Re: Jira contributor access
Hi Güven, I have added you to the contributor role. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 13, 2020, at 6:09 AM, Güven Cenan Güvenal wrote: > > Hello, > > I want Jira contributor access for my account “guvencenan”. > > Thanks in advance. >
[ANNOUNCE] Apache NiFi 1.11.1 vulnerability announcements
Apache NiFi Community, The https://nifi.apache.org/security.html <https://nifi.apache.org/security.html> page has been updated with 1 vulnerability discovered in previous NiFi versions which has been resolved in release 1.11.1. The severity of this issue was determined to be ‘important'. Questions about this vulnerability can be directed to secur...@nifi.apache.org <mailto:secur...@nifi.apache.org>. CVE-2020-1942: Apache NiFi information disclosure in logs Severity: Important Versions Affected: Apache NiFi 0.0.1 - 1.11.0 Description: The flow fingerprint factory generated flow fingerprints which included sensitive property descriptor values. In the event a node attempted to join a cluster and the cluster flow was not inheritable, the flow fingerprint of both the cluster and local flow was printed, potentially containing sensitive values in plaintext. Mitigation: Implemented Argon2 secure hashing to provide a deterministic loggable value which does not reveal the sensitive value. Users running any previous NiFi release should upgrade to the latest release. Released: February 4, 2020 If you identify new security issues within the NiFi 1.11.1 release, please forward your report to secur...@nifi.apache.org <mailto:secur...@nifi.apache.org> and do not disclose the issue publicly. The security vulnerability reporting and disclosure process can be found here: https://www.apache.org/security/committers.html <https://www.apache.org/security/committers.html>. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
Re: Monitoring/Alerting
Have you tried using the MonitorActivity [1] processor? "Monitors the flow for
activity and sends out an indicator when the flow has not had any data for some
specified amount of time and again when the flow's activity is restored."
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.MonitorActivity/
<https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.1/org.apache.nifi.processors.standard.MonitorActivity/>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 7, 2020, at 11:26 AM, Emanuel Oliveira wrote:
>
> Hi,
>
> Simple suggestion to detect lag would be (imagining 1 single flow handling
> the 15 inbound streams):
>
> *Processor UpdateAttribute*
> Store State= store state locally
>
> add one attribute per stream to keep last ts
> last_stream_1_ts if stream=1 then ${now:toNumber()} else ${getStateValue("
> last_stream_1_ts")}
>
>
> processor RouteOnAttribute
> stream1_lag if ${now:toNumber()} - ${last_stream_1_ts} > 8640 then
> 'yes' else 'no'.
>
>
> Sorry i used pseudo code -im in a restaurant :) - but just wanted to share
> how you generate Boolean attributes that one can then use to do whatever
> action needed.
>
> Hope this helps,
> Emanuel O.
>
>
>
>
>
>
>
>
>
>
>
> On Fri 7 Feb 2020, 18:47 Marroquin, Stuardo X. -ND, <
> stuardo.x.marroquin@disney.com
> <mailto:stuardo.x.marroquin@disney.com>> wrote:
>
>> Good afternoon,
>>
>>
>>
>> I was wondering if there was a way to monitor the data flow into NiFi and
>> also from NiFi to our storage device.
>>
>> We have approximately 15 streams of files coming into NiFi at a rate of a
>> file per minute, 24/7, 365 days, varying in size from 4MB to 12MB each. We
>> need a way to ensure that we are notified if one of those streams stops.
>>
>> I saw that you have PutEmail but there is not a whole lot of documentation
>> on setting it up and I don’t even know if that will meet our needs.
>>
>> Any help you can provide on this would be greatly appreciated.
>>
>>
>>
>> Kind Regards,
>>
>> Stu Marroquin
>>
>> [image: blue_strip]
>>
>> *Stu Marroquin*
>>
>> Information Management Senior Specialist
>>
>> The Walt Disney Account
>>
>> Atos North America
>>
>> 407-619-2498
>>
>> stu.marroq...@atos.net
>>
>> (DISC) 5601 Center Drive
>> Orlando, FL 32830
>>
>> www.na.atos.net
>>
>>
>>
>> [image: Atos]
>>
>>
>>
>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive this
>> e-mail in error, please notify the sender immediately and destroy it. As
>> its integrity cannot be secured on the Internet, the Atos group liability
>> cannot be triggered for the message content. Although the sender endeavors
>> to maintain a computer virus-free network, the sender does not warrant that
>> this transmission is virus-free and will not be liable for any damages
>> resulting from any virus transmitted
Re: Request for contributor access
Hi Lars, I added you to the contributor role. Thanks for your work. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 7, 2020, at 10:50 AM, Lars Winderling > wrote: > > Hi nifi devs, > > please grant me contributor access. My username on issues.apache.org is > kahalemakai. > I have been working on 2 issues (for a start), and I hope to keep that > going. > In case this information is insufficient, please tell me. > > Thank you in advance! > Best, > Lars >
Re: NiFi Site-to-Site error [SEC=UNCLASSIFIED]
Hi Ben, From the error, it appears the HTTPS handshake (to identify the endpoint) is successful but the communication over the RAW transport is failing. Can you share the approximate size of your flowfiles being transmitted between the two endpoints? If the contents are large, it may require increasing the timeout to a value like 300 seconds to allow the socket transmission to complete. You may also want to investigate changing the transport protocol from RAW to HTTP, which will allow for slightly different communication (RAW = HTTP(S) handshake, socket transport; HTTP = HTTP(S) for both). Do you have any network traffic monitoring (Wireshark, etc.) where you can evaluate the actual data being transmitted between the nodes? If you see data transfer simply not completing by the time the timeout occurs, you basically have two options - raise the timeout or perform a split on the content within the transmitting NiFi and reassemble at the receiving end. Koji Kawamura put some detailed diagrams explaining the various transport protocols on the wiki [1]. Hopefully this helps or at least leads you down the right path. With additional details we may be able to offer further assistance. [1] https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP%28S%29+as+a+transport+mechanism+for+Site-to-Site <https://cwiki.apache.org/confluence/display/NIFI/Support+HTTP(S)+as+a+transport+mechanism+for+Site-to-Site> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 6, 2020, at 4:22 PM, Fullarton, Ben MR > wrote: > > UNCLASSIFIED > > Hi Apache NiFi Team, > > Hoping you can help with an error Im getting with our NiFi1.8.0 Site-to-Site > running over a low bandwidth network. > > Basically we are testing the performance of NiFi over poor networks . Our 2 > NiFi servers are built on Windows 2016 servers with site-to-site > communications enabled and secured with SSL certificates. We have configured > the servers as follows; > > Sender (Client) NiFi Server > OS: Windows Server 2016 standard > NiFi Version: 1.8.0 > NiFi.Properties: > # Site to Site properties: > nifi.remote.input.host=senderhostname > nifi.remote.input.secure=true > nifi.remote.input.socket.port= > nifi.remote.input.http.enabled=true > nifi.remote.input.http.transaction.ttl=600 sec > nifi.remote.contents.cache.expiration=10 secs > Remote Process Group details: > URLs: https://Destinationhostname:8443/nifi/ > Transport Protocol: RAW > Communication Timeout: 60 sec > Yield Duration: 1 sec > > Destination (Server) NiFi Server > OS: Windows Server 2016 standard > NiFi Version: 1.8.0 > NiFi.Properties: > # Site to Site properties: > nifi.remote.input.host=destinationhostname > nifi.remote.input.secure=true > nifi.remote.input.socket.port= > nifi.remote.input.http.enabled=true > nifi.remote.input.http.transaction.ttl=600 sec > nifi.remote.contents.cache.expiration=10 secs > > > So far, sending files between the servers via remote port group is fine under > normal conditions. However when we restrict the network link throughput > (channel capacity) via our network emulator (iTrinegy INE) to 100kbps the > NiFi client (sender) fails to transfer data and provides the following error > message [sensitive details have been reacted]; > > RemoteGroupPort[name=senderhostname,targets=https://destinationhostname.fqdn:8443/nifi/] > failed to communicate with remote NiFi instance due to java.io.IOException: > Failed to confirm transaction with Peer[url=nifi://destinationhostname:] > due to java.net.SocketTimeoutException: Timed out reading from socket > connected to destinationhostname: > > Consequently, the files stop transmitting to the destination NiFi server. > > Can you please advise on how we can fix this error whilst maintaining a > network link throughput (channel capacity) less than or equal to (≤) 100 > Kbps? Perhaps we may be able to change some aspect of the Nifi configuration > or underlying configuration files. > > If you are able to provide any additional guidance about configuring NiFi to > work over low throughput, high latency and high packet loss links it would be > greatly appreciated. > > If not able to provide us advice, are you able to suggest someone who may > assist us? > > Many thanks > > > Kind Regards, > > Ben Fullarton > Systems Engineer > Contractor to Defence > Ph: 02 6265 0158 > Mob: 0427 800 745 > email: ben.fullar...@defence.gov.au<mailto:ben.fullar...@defence.gov.au> > IMPORTANT: This email remains the property of the Department of Defence. > Unauthorised communication and dealing with the information in the email may > be a serious criminal offence. If you have received this email in error, you > are requested to contact the sender and delete the email immediately. > >
Re: Potential 1.11.X showstopper
I have no input on the specific issue you’re encountering, but a pattern we have seen to reduce the overhead of multiple remote input ports being required is to use a “central” remote input port and immediately follow it with a RouteOnAttribute to distribute specific flowfiles to the appropriate downstream flow / process group. Whatever sends data to this port can use an UpdateAttribute to add some “tracking/routing” attribute on the flowfiles before being sent. Inserting Merge/Split will likely affect your timing due to waiting for bins to fill, depending on your volume. S2S is pretty good at transmitting data on-demand with low overhead on one port; it’s when you have many remote input ports that there is substantial overhead. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 6, 2020, at 2:34 PM, Elli Schwarz > wrote: > > We ran that command - it appears the site-to-sites that are causing the > issue. We had a lot of remote process groups that weren't even being used (no > data was being sent to that part of the dataflow), yet when running the lsof > command they each had a large number of open files - almost 2k! - showing > CLOSE_WAIT. Again, there were no flowfiles being sent to them, so can it be > some kind of bug that keeping a remote process group open is somehow opening > files and not closing them? (BTW, the reason we had to upgrade from 1.9.2 to > 1.11.0 was because we had upgraded our Java version and that cause an > IllegalBlockingModeException - is it possible that whatever fixed that > problem is now causing an issue with open files?) > > We now disabled all of the unused remote process groups. We still have > several remote process groups that we are using so if this is the issue it > might be difficult to avoid, but at least we decreased the number of remote > process groups we have. Another approach we are trying is a merge content > before we send to the Nifi having the most issues, to have fewer flow files > sent at once site to site, and then splitting them after they are received. > Thank you! > >On Thursday, February 6, 2020, 2:19:48 PM EST, Mike Thomsen > wrote: > > Can you share a description of your flows in terms of average flowfile size, > queue size, data velocity, etc.? > Thanks, > Mike > > On Thu, Feb 6, 2020 at 1:59 PM Elli Schwarz > wrote: > > We seem to be experiencing the same problems. We recently upgraded several > of our Nifis from 1.9.2 to 1.11.0, and now many of them are failing with "too > many open files". Nothing else changed other than the upgrade, and our data > volume is the same as before. The only solution we've been able to come up > with is to run a script to check for this condition and restart the Nifi. Any > other ideas? > Thank you! > > On Sunday, February 2, 2020, 9:11:34 AM EST, Mike Thomsen > wrote: > > Without further details, this is what I did to see if it was something > other than the usual issue of having not enough file handlers available. > Something like a legitimate case of someone forgetting to close file > objects or something in the code itself. > > 1. Setup a 8core/32GB VM on AWS w/ Amazon AMI. > 2. Pushed 1.11.1RC1 > 3. Pushed the RAM settings to 6/12GB > 4. Disabled flowfile archiving because I only allocated 8GB of storage. > 5. Setup a flow that used 2 generateflow instances to generate massive > amounts of garbage data using all available cores. (All queues were setup > to hold 250k flow files) > 6. Kicked it off and let it run for probably about 20 minutes. > > No apparent problem with closing and releasing resources here. > > On Sat, Feb 1, 2020 at 8:00 AM Joe Witt wrote: > >> these are usually very easy to find. >> >> run lsof -p pid. and share results >> >> >> thanks >> >> On Sat, Feb 1, 2020 at 7:56 AM Mike Thomsen >> wrote: >> >>> >>> >> https://stackoverflow.com/questions/59991035/nifi-1-11-opening-more-than-50k-files/60017064#60017064 >>> >>> No idea if this is valid or not. I asked for clarification to see if >> there >>> might be a specific processor or something that is triggering this. >>> >> >
Re: View Usage brings up (default) Apache NiFi Overview documentation
Russ, I don’t know where the idea to do a “single-module project” came from. The recommended (read: working) method is to have multiple Maven modules — a parent (usually called “nifi-myprocessor-bundle”), a processor (“nifi-myprocessor-processor”), and a NAR builder (“nifi-myprocessor-nar”). This is how the Maven NIFi archetype structures a generated custom processor project [1]. Similarly, my presentation on building custom processors addresses this structure the same way [2], as does the official Developer Guide [3]. I briefly reviewed the contents of your notes, and they are nicely formatted. It appears the advantage of the “single-module” approach is a slightly cleaner presentation in your IDE, while the downside is that the processor annotations are not evaluated completely (and potentially other problems; I have not investigated setting up this non-supported structure further). I would guess that the problem is in a single-module project, the packaging type is NAR, so the bundling process of building the NAR from a compiled and built JAR doesn’t fully occur. Your notes don’t show the results of building the single module, and the contents of the target/ directory to be specific. I don’t mean to sound glib, but I don’t have the bandwidth at this time to pursue the single-module structure further, as I know of no one recommending it and have no idea why it would work other than by accident. We try to avoid recommending additional complexity unless it is required, and to the best of my knowledge, the “3 module” approach is the minimum required for successful (and complete) custom processor building and deployment. Hope this helps. [1] https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions <https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions> [2] https://github.com/alopresto/slides/tree/master/dws_barcelona_2019 <https://github.com/alopresto/slides/tree/master/dws_barcelona_2019> [3] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 28, 2020, at 1:49 PM, Russell Bateman wrote: > > Andy, > > I have formulated, on the basis of posts I've seen over the years, that there > are two ways to build a NAR containing one's own, custom processors, what I > term a /single-module product/ and a /multi-module product/. mostly based on > whether I need multiple modules in IntelliJ IDEA or only one. I expose this > in a page of notes that I have maintained for some time and I just looked at > the single-module version, the one with the simple NAR build and discovered > (I had probably just never noticed before) that the simple one suffers from > the same problem as I suffer from now, to wit, that the simpler project > structure and /pom.xml/ files yields a processor whose > /@CapabilityDescription/ is ignored. > > My original projects used the more complicated project build, but I saw > someone propose the simpler one, tried it, it worked, and I have been trying > to adopt it for my new project where I plan for only one module, i.e.: the > project itself, and therefore dispence with separate /nar/ and /processor/ > subdirectories. > > This is exposed at https://www.javahotchocolate.com/notes/nifi-project.html. > > If you have time and choose to look at it, and the /single-module project/ > leaps out at you with the answer (as to why it can't be done that way or > what, of this way, prevents the documentation from working), would you point > that out? I have rebuilt using the more complex build process with /nar/ > subdirectory and three instead of one /pom.xml/ files and I now get the > documentation to work. This is the answer, but why can't I build using one > /pom.xml/ and the simpler project structure? > > Russ > > On 1/28/20 10:33 AM, Russell Bateman wrote: >> Andy, >> >> The processor is really called NoOp (not CustomProcessor), it's a straight, >> pass-through that doesn't copy let alone modify the flowfile. It's used for >> debugging and crafting flows (and is always ultimately discarded or replaced >> with a real processor doing real things). >> >> In my new NAR, I see >> >>META-INF >> bundled-dependencies >> docs >> maven >>com.windofkeltia.pipeline >> pom.properties >> pom.xml >> services >>*org.apache.nifi.processor.Processor* (of which, >>com.windofkeltia.processor.NoOp) >>com >> windofkeltia >>processor >&g
Re: View Usage brings up (default) Apache NiFi Overview documentation
Hi Russ,
I hate to suggest something so simple, but have you cleared the cache of your
browser? Sometimes this can be a result of stale caching.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jan 27, 2020, at 2:30 PM, Russell Bateman wrote:
>
> Addendum: the @TagsI add are not showing up in the tag field at the left, but
> when I scroll down in the processor list and see my new custom processor, I
> do see these tags listed next, to its right.
>
> On 1/27/20 3:27 PM, Russell Bateman wrote:
>> Perhaps I made my question less clear that I could have. Consider that I
>> have this annotation on the class of processor:
>>
>>@CapabilityDescription( "Custom processor usage statement..." )
>>
>> and, because my processor is listed in the file
>>
>>/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor/
>>
>> as
>>
>>/com.windofkeltia.processor.//CustomProcessor/
>>
>> I am able to create a perfectly running and working instance of this
>> processor on the canvas through the web browser UI of NiFi 1.10.0.
>>
>> However, when I right-click the processor instance and choose View usage, I
>> do not see documentation with
>>
>>"Custom processor usage statement..."
>>
>> but, instead, I see what I assume is default page content saying
>> principally, "Apache NiFi Overview". I am looking for help/things to check
>> that are keeping NiFi from displaying documentation for my capability
>> description, properties, relationships, etc. Also, I am using the @Tags
>> annotation to add a couple of quick-find tags: neither of these are showing
>> up when I go to create an instance of the processor. This is also likely a
>> clue to what I'm doing wrong.
>>
>> Thanks,
>> Russ
>>
>>
>> On 1/24/20 3:30 PM, Russell Bateman wrote:
>>> My custom processor's usage, which should come from the
>>> @CapabilityDescription annotation of the class containing the onTrigger()
>>> method, nevertheless is nothing more than the Apache NiFi Overview. I am
>>> able to place an instance on the canvas via the processor Component Tool
>>> (so, the custom processor is there--I just can't get my usage statement):
>>>
>>>@SupportsBatching
>>>@SideEffectFree
>>>@InputRequirement( InputRequirement.Requirement.INPUT_REQUIRED )
>>>*@CapabilityDescription( "Custom processor usage statement..**." )*
>>>public class CustomProcessor extends AbstractSessionFactory Processor
>>>{
>>> @Override
>>> public void onTrigger( final ProcessContext context, final
>>>ProcessSessionFactory sessionFactory )
>>> throws ProcessException
>>> {
>>>...
>>> }
>>>
>>> etc.
>>>}
>>>
>>> What thing missing should I be looking for to fix this?
>>>
>>> I'm sure it's some stupid oversight that I have not failed to provide in
>>> other (successfully documented) custom processors I've written. I have
>>> compared this processor to those others; I've structured it identically,
>>> etc. and I have pored through
>>> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#documenting-a-component
>>> a number of times. Usage just doesn't work.
>>>
>>> Thanks.
>>>
>>>
>>
>
Re: Any work planned for adding Kerberos pkinit support?
I believe the issue here is not NiFi to NiFi communication (where we use PKI certs for TLS mutual authentication) or user to NiFi (where client certificate authentication is currently available), but for NiFi to use a PKI cert to retrieve a TGT & service ticket on-demand rather than persisting a keytab internally. This allows for better control over the credential material via X.509 expiration & revocation. I know there are people working on this that are currently investigating pkinit to see if it is feasible to implement. The ticket mentioned below is for complementing keytab capability with username/password credentials for a related but different use case. Paul, it sounds like your move to this approach, while challenging, will definitely enhance the security and auditability of your system. We would definitely encourage you to contribute back your development process. I think the cleanest approach would probably be to provide a custom implementation of a controller service like KerberosCredentialsService [1] called PkinitCredentialsService which points to a certificate/keystore and accepts a password. You can reference KeytabCredentialsService [2] as a reference implementation. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-kerberos-credentials-service-api/src/main/java/org/apache/nifi/kerberos/KerberosCredentialsService.java#L22 <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-kerberos-credentials-service-api/src/main/java/org/apache/nifi/kerberos/KerberosCredentialsService.java#L22> [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-kerberos-credentials-service-bundle/nifi-kerberos-credentials-service/src/main/java/org/apache/nifi/kerberos/KeytabCredentialsService.java <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-kerberos-credentials-service-bundle/nifi-kerberos-credentials-service/src/main/java/org/apache/nifi/kerberos/KeytabCredentialsService.java> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 24, 2020, at 7:51 AM, Joe Witt wrote: > > Hello > > We already have tons of support for PKI/mutual Auth connections. For the > kerberos stuff where you'd rather supply username/pword than keytabs we got > you there too based on the JIRA highlighted elsewhere in this thread. > > Are you seeing/concerned about other gaps? > > Thanks > > On Fri, Jan 24, 2020 at 10:37 AM Paul Poulosky > wrote: > >> Hi, Shawn. >> >> This is a corporate mandate in order to migrate to best practice w.r.t. >> security. >> >> Yahoo runs very large 5000+ node clusters (Primarily HDFS / HBase / Hive / >> Spark / Storm) that are multi-tenant. The worker processes run as >> individual headless users per project / application. A given project had >> been using keytabs on launcher boxes in order to get a TGT into their >> workers that would allow that headless user to authenticate with services. >> Keytabs don't expire, and there is no way of tracking who has a copy. We >> have to depend on our users to set the file permissions up properly to not >> have them get stolen, etc. >> >> Moving to PKinit and X509 certs (which expire after 30 days) gives us a >> higher level of security in our environment. We have to do this everywhere, >> not just Nifi. >> >> We would prefer to hack up the processors in order to do this work. We'd >> like to do it in a way that would be acceptable to push back to open >> source. >> >> Any direction you could give us in meeting this objective would be greatly >> appreciated. >> >> Thanks, >> Paul >> >> On Thu, Jan 23, 2020 at 4:52 PM Shawn Weeks >> wrote: >> >>> If you don’t mind, what is the use case where keytabs aren’t working for >>> you? I’ve always found them easier since I don’t think you can setup >>> passwords for service principals only user principals. >>> >>> >>> >>> Thanks >>> >>> Shawn >>> >>> >>> >>> *From: *Pat White >>> *Date: *Thursday, January 23, 2020 at 4:46 PM >>> *To: *Shawn Weeks >>> *Cc: *"dev@nifi.apache.org" , Paul Poulosky < >>> ppoul...@verizonmedia.com> >>> *Subject: *Re: Any work planned for adding Kerberos pkinit support? >>> >>> >>> >>> Wow, a lot of work on this! Thank you very much Shawn, and my apologies >>> for completely missing the Jiras, thanks for the help. >>> >>>
Re: [VOTE] Release Apache NiFi MiNiFi C++ 0.7.0 (RC1)
+1 (binding) I encountered a number of obstacles during this validation, some of which are definitely because I haven’t worked on the C++ effort (or even built it) in at least 6 months. These are not complaints, just wrinkles I captured so we can improve this process for next time. Noted here for posterity: * The SHA-512 digest I calculated locally matched the email, but the .sha512 files are not published in the hosted server for any artifacts. * The GPG verification command in the helper email should include -v to show the underlying digest algorithm used * The wget commands have newlines inserted, which means they cannot be copied/pasted into the terminal without manual modification * The README section on bootstrapping helped immensely, but is not perfectly aligned with the current process * The wording of “disable tests…..disabled” is unclear (in this case it means that the tests will be run) * The bootstrap instructions are missing the directive to change into the build/ directory before running make (directory in example prompt is wrong) * The bootstrap instructions are missing the directive to untar the binary artifact and change into that directory before running ./bin/minifi.sh start * A sample flow (config.yml) with a GenerateFlowFile and LogAtttribute flow would be helpful to allow people to verify the successful install of MiNiFi without going through the full flow design in NiFi, export template, convert template process All in all, a lot of great work done on this release. Thanks Arpad. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 10, 2020, at 12:55 PM, Matt Gilman wrote: > > +1 (binding) > > Ran through release helper. Verified signature, checksums, build, tests, > etc. Looks good. > > Thanks for RMing Arpad! > > On Fri, Jan 10, 2020 at 3:39 PM Kevin Doran wrote: > >> +1 (binding) >> >> Ran through the steps in the release helper guide. Noticed a few >> release helper instructions that can be improved for next time (no >> mention of bootstrap.sh), >> but was able to figure it out by using the README file included in the >> source release. >> >> Hashes & sig all looked good. Was able to build, including the docker >> image. Tests >> all passed. Tested integration with a NiFi cluster with the 0.7.0 >> minifi sending flow files over s2s and everything worked as expected. >> >> Overall, seems to be a very solid, polished release. Nice work all! >> >> On Fri, Jan 10, 2020 at 3:35 PM Joe Witt wrote: >>> >>> +1 (binding) >>> >>> Comments >>> - Notice file copyright year needs to be 2020 >>> - I did not enable python support but I get the minifi-python dir anyway >>> and log entry on startup such as >>> Caught Exception ModuleNotFoundError: No module named 'google' >>> >>> Great progress with this release! Nice work. Building and running tests >>> was easier than I ever remember. >>> >>> Thanks >>> >>> On Fri, Jan 10, 2020 at 8:27 AM Dániel Bakai >>> wrote: >>> >>>> +1, non-binding >>>> >>>> Verifications performed: >>>> >>>> SHA256 checksums downloaded: OK >>>> SHA512 checksum of nifi-minifi-cpp-0.7.0-source.tar.gz sent in mail: OK >>>> GPG signatures downloaded: OK >>>> Git tag in email matches sources in >> nifi-minifi-cpp-0.7.0-source.tar.gz: OK >>>> >>>> bootstrap.sh with default options && make && make package && sudo make >> test >>>> ARGS="-j4 --output-on-failure" && run package with a GenerateFlowFile >> -> >>>> LogAttribute flow: >>>> macOS 10.14.6: TailFileTests sometimes fail, otherwise OK >>>> ubuntu:bionic docker container: had to install 'tzdata' package for >>>> ExpressionLanguageTest to work, CTailFileChunkTests and >>>> CTailFileDelimitedTests consistently fail when ran in parallel, running >>>> them sequentially works, otherwise OK >>>> centos:7 docker container: had to install 'zlib-devel' and 'which', >>>> otherwise OK >>>> debian:stretch docker container: OK >>>> fedora:29 docker container: had to install 'zlib-devel' and 'which', >>>> otherwise OK >>>> ubuntu:xenial docker container: had to install 'tzdata' package for >>>> ExpressionLanguageTest to work, otherwise OK >>>> >>>> Run appropriate convenience binary with a GenerateFlowFile -> >> LogAttribute >>>> flow: &
Re: Jira Contributor Access
H Sivanath, I added you to the contributor role. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 7, 2020, at 6:20 AM, Nagasivanath Dasari > wrote: > > Dear Community, > > My JIRA username is “nagasivanath”, I would like to contribute towards > apache nifi development, please provide me the access. > > Regards, > Sivanath
Re: Jira contributor access
Tushar, I have added you to the contributor role. Most people write their articles on their own blog and post it wherever they like (a privately-owned site, Medium, DZone, GitHub Pages, vendor sites, etc.). For the official Apache NiFi blog, I believe you need to be a PMC member to post there. There is no formal review process because you can write whatever you like. If you would like constructive feedback, you can send it to this mailing list requesting that. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 6, 2020, at 6:45 PM, Tushar S wrote: > > Hi Andy > Sorry , it’s “tuhsar120” > > Can you also please guide me how to submit a blog /how to do > article for review and then publish. > > Regards > Tushar > > Sent from my iPhone > >> On 07-Jan-2020, at 7:27 AM, Andy LoPresto wrote: >> >> Hi Tushar, >> >> The username you provided does not exist on issues.apache.org >> <http://issues.apache.org/>. I also tried “tushar120120” and that did not >> exist either. Please double check you’re username. >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Jan 6, 2020, at 4:05 AM, Tushar S wrote: >>> >>> Hi >>> Kindly give me Jira contributor access, my Jira username is tuhsar120120 >>> >>> Regards >>> Tushar >>> >>> Sent from my iPhone >>> >>>> On 06-Jan-2020, at 5:19 PM, Tushar S wrote: >>>> >>>> Hi >>>> Kindly give me Jira contributor access, my Jira username is tuhsar120120 >>>> >>>> Regards >>>> Tushar >>
Re: Adding disclaimer to Log-In Page
Hi Vamsi, You will have to add your desired disclaimer text to login.jsp [1] and then rebuild the application. You can also set custom text in nifi.properties for the property nifi.ui.banner.text which will be displayed on all pages. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/pages/login.jsp <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/pages/login.jsp> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 6, 2020, at 2:22 AM, Vamsi.krishna.ts > wrote: > > Hi Team, > I have a requirement to add a Disclaimer to NiFi Log-in page, but I am not > able to find where exactly I have to do changes. > Could you please help me, on how and where to do the changes. > > -Vamsi > > Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.
Re: Jira contributor access
Hi Kristjan, I have added you to the contributor role. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 6, 2020, at 2:22 PM, Kristjan Antunovic > wrote: > > Jira username: superkool
Re: Jira contributor access
Hi Tushar, The username you provided does not exist on issues.apache.org <http://issues.apache.org/>. I also tried “tushar120120” and that did not exist either. Please double check you’re username. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jan 6, 2020, at 4:05 AM, Tushar S wrote: > > Hi > Kindly give me Jira contributor access, my Jira username is tuhsar120120 > > Regards > Tushar > > Sent from my iPhone > >> On 06-Jan-2020, at 5:19 PM, Tushar S wrote: >> >> Hi >> Kindly give me Jira contributor access, my Jira username is tuhsar120120 >> >> Regards >> Tushar
Re: Jira contributor access
Hi Josh, You’ve been added as a contributor. Enjoy. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 19, 2019, at 3:49 PM, Phelps, Joshua E [US] (MS) > wrote: > > Hi. My username is JoshE and I'd like to request Jira contributor access for > the Apache Nifi project. > > Thanks, > > Josh >
Re: 【New Feature】Connector to Huawei Cloud
Hi David, Welcome to the community. You should start by reading the Contributor Guide [1] which explains the process of contributing to the project, and then the Developer Guide [2] which has resources and instructions on writing code for NiFi. [1] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> [2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 19, 2019, at 2:02 PM, David Sanchez Plaza > wrote: > > Hello all! > > This is my first message, so my apologies if done something wrong. > > I will like to extend the Apache NiFi capabilities to read/write to Huawei > Cloud. I am starting to read about the development process. > > I also have direct access to R team, what would potentially help on the > development. > > If there is any guide/document/advice you consider basic, please, mind > sharing with me? > > If someone has experience and want to collaborate, also, feel welcome :) > > Thanks a lot, Happy Nifying! >
Re: Cluster issue
Hi Phil, Are you sure that the certificates have valid SubjectAlternativeName entries for the proper hostnames? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Dec 3, 2019, at 7:53 PM, Phil H wrote: > > Okay thanks I’ll check that state/local thing out. > > Drilling a little further on the main issue, it seems to boil down to NiFi > on Server B having an UnknownHostException for the FQDN for Server B. I am > just using /etc/hosts for the naming (no DNS in our server enclave) and of > course I have confirmed spelling, etc. I even wrote a two line Java program > that calls InetAddress.getAllByName, and passing it the same FQDN (copies > and pasted from the nifi stack trace) resolves without a problem. So now I > am really confused. I have rebooted Server B and that didn’t change > anything. I also tried an alternate FQDN and that has the same behavior on > Server B > > Really confused now! > > On Wed, 4 Dec 2019 at 01:14, Bryan Bende wrote: > >> Hello, >> >> To get rid of the old values from before securing your cluster, remove >> the state/local directory on both servers (assuming you don't have any >> processor state that you care about since this is a new cluster). >> >> For the other issue, is there a stacktrace with more info? >> >> Thanks, >> >> Bryan >> >> On Mon, Dec 2, 2019 at 7:14 PM Phil H wrote: >>> >>> Hi there, >>> >>> I (almost) have a secure nifi 1.9.2 cluster of two servers. Server A >> starts >>> up fine, but Server B fails to start with this error (I cannot copy/paste >>> as it is an offline system) >>> >>> Could not start listening for incoming connections in order to load >> balance >>> data across the cluster. Please verify the values of >>> ‘nifi.cluster.load.balance.port’ and ‘ nifi.cluster.load.balance.host’ >>> property as well as the ‘nifi.security.*’ properties >>> >>> The config between the two servers are identical, apart from the >> respective >>> host names, and there are no other applications trying to bind on any of >>> the ports on Server B. I have also tried changing ports in case it was >>> something simple like that. >>> >>> Additionally, when restarting Server A, I occasionally see references to >>> cluster members from before I had the secure config (eg: IP address / >> port >>> 80) - how do I expunge any prior info regarding cluster members and start >>> again with the two servers I now have?? >>> >>> >>> Regards, >>> Phil >>
Re: Add me as contributor
I’ll take care of this Grant. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 27, 2019, at 21:48, Grant Henke wrote: > > Can someone please add me, username granthenke, as a jira contributor? I > would like to assign myself a few Kudu related jiras such > as NIFI-6867, NIFI-6895, and NIFI-6908. > > Thank you, > Grant
Re: Issues while configuring nifi cluster
The message about the blank sensitive properties key is important and is informing you that the sensitive configuration values are not being protected because that key has not been set. However, it is not blocking any operations from occurring. The subsequent messages are telling you that the Flow Registry does not contain the specific version (version 2) of one of the flows being used, so the node cannot sync with it. Then it reports that the node cannot join the cluster because the local flow is not the same as the cluster flow. If you have made local changes and want to keep them, you’ll have to perform manual synchronization on the cluster nodes to duplicate those changes. If not, you can delete the flow.xml.gz file from the local node’s conf/ directory and after restarting, the node will inherit the cluster flow. For node 2, the same message as the beginning of node 1 is occurring. You should check that your Flow Registry is configured correctly and has the most recent version of the flow under version control. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 4, 2019, at 7:27 AM, Venkata Raghava Raju Bhupathiraju > wrote: > > Hello Andy, > > Thanks for you response, Please find the below errors in the log file from > both the Nifi Cluster servers, > > Nifi - 1 > > 2019-11-04 14:04:51,070 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > *A blank sensitive properties key was provided > * > 2019-11-04 14:04:51,070 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * Specify a unique key in nifi.properties > * > 2019-11-04 14:04:51,070 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * for nifi.sensitive.props.key > * > 2019-11-04 14:04:51,071 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * > * > 2019-11-04 14:04:51,071 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > *The Encrypt Config Tool in NiFi Toolkit can be used to > * > 2019-11-04 14:04:51,071 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * migrate the flow to the new key > * > 2019-11-04 14:04:51,071 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > > 2019-11-04 14:04:56,455 ERROR [Framework Task Thread Thread-1] > o.a.nifi.groups.StandardProcessGroup Failed to synchronize > StandardProcessGroup[identifier=b066b84a-016d-1000--af957491] with > Flow Registry because could not retrieve version 2 of flow with identifier > 091ba32b-270a-43e5-9459-2e56e2650950 in bucket > fdaf7347-5f0a-4a8b-90f6-a2ffbd1e4636 > 2019-11-04 14:09:58,043 ERROR [Reconnect to Cluster] > o.a.nifi.controller.StandardFlowService Handling reconnection request failed > due to: org.apache.nifi.controller.UninheritableFlowException: Failed to > connect node to cluster because local flow is different than cluster flow. > 2019-11-04 14:09:58,072 ERROR [Reconnect to Cluster] > o.a.n.c.c.node.NodeClusterCoordinator Event Reported for 10.1.1.14:8081 > <http://10.1.1.14:8081/> -- Node disconnected from cluster due to > org.apache.nifi.controller.UninheritableFlowException: Failed to connect node > to cluster because local flow is different than cluster flow. > > Nifi - 2 > > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > *A blank sensitive properties key was provided > * > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * Specify a unique key in nifi.properties > * > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * for nifi.sensitive.props.key > * > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * > * > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > *The Encrypt Config Tool in NiFi Toolkit can be used to > * > 2019-11-04 14:04:50,672 ERROR [main] org.apache.nifi.encrypt.StringEncryptor > * migrate the flow to the new key > * > 2019-11-04
Re: Issues while configuring nifi cluster
Images do not come through consistently on this email list, so please host that elsewhere and link to it, or copy/paste the content of it. Without seeing the image, I am not positive, but based on the other text in your email, I suspect your nifi.sensitive.props.key value is empty in nifi.properties. This is not a blocker but is strongly discouraged. The second issue is likely a malformed XML file in users.xml or authorizers.xml. It could also be the flow.xml.gz file. Without additional debugging information (further stack trace, those XML files), we will not be able to help more than that. If you can share those, please do. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 31, 2019, at 1:24 PM, Venkata Raghava Raju Bhupathiraju > wrote: > > Hello, > > I am getting the below while configuring nifi-cluster, > > A blank sensitive properties key was provided. Specify a unique key in > nifi.properties for nifi.sensitive.props.key > > > > We are getting the below exception as well, > > Caused by: org.springframework.beans.factory.BeanCreationException: Error > creating bean with name 'flowService': FactoryBean threw exception on object > creation; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name 'flowController': FactoryBean threw exception on object creation; > nested exception is java.lang.ArrayIndexOutOfBoundsException: 1 > > Thanks, > Raghava Raju > Softility, Inc. | E: vbhupathir...@softilityinc.com > <mailto:vbhupathir...@softilityinc.com>
Re: [ANNOUNCE] New Apache NiFi Committer Peter Turcsanyi
Congratulations Peter. Looking forward to your continued contributions. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 28, 2019, at 9:39 AM, Matt Burgess wrote: > > Congratulations Peter! > > On Sun, Oct 27, 2019 at 9:05 PM Aldrin Piri wrote: >> >> Apache NiFi community, >> >> On behalf of the Apache NiFI PMC, I am very pleased to announce that Peter >> has accepted the PMC's invitation to become a committer on the Apache NiFi >> project. We greatly appreciate all of Peter's hard work and generous >> contributions to the project. We look forward to continued involvement >> in the project. >> >> Peter has provided several new extensions and improvements enhancing NiFi's >> interoperability with cloud services. He has also found and remedied >> several bugs, is a regular participant in code reviews and an active >> presence on the mailing lists helping out the community. >> >> Welcome and congratulations!
Re: Re: [Discuss] Data prioritization - A proposed solution
This is an interesting idea to solve an admitted problem, but I wonder how it comports with the core tenets of Flow Based Programming on which NiFi is modeled. This seems to introduce globally-coupled dependencies between all queues in a flow, where another solution (flow segment-based resource allocation) might solve this problem without requiring per-queue contention on every cycle. I think with the stateless NiFi work there has been some discussion around being able to control the resource allocation for a flow segment. Sam Hjemfelt, any thoughts here? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 17, 2019, at 8:54 AM, Kessler, Jon wrote: > > Joe, hopefully I addressed all of your questions: > > Your interpretation of the scheduling aspect is correct. These queues will > pretend to be empty a certain % of the time if higher priority data recently > moved elsewhere. That % is configurable on a per rule basis which allows the > operator to determine how much to stagger the data associated with each rule. > That % is also how the rules are ranked in terms of order of priority. The > higher the %, the more often a rule will make use of its threads so the > higher its priority is considered to be. > > Administration: You are correct that the ruleset is provided at the flow > controller level but will be leveraged by all connections regardless of > whether or not they use the BucketPrioritizer (more details on this below). > This overall solution only works if all FlowFileQueues are of this new > implementation which is why I tied it to nifi.properties settings. > > The sorting function here takes place on insertion into any connection on > which a BucketPrioritizer is set. Once a FlowFile has been sorted into a > bucket we maintain that state so that each time it moves into a new > connection we already know in which bucket it should be placed without > needing to have a BucketPrioritizer set on that connection. Each bucket in a > connection is just a FIFO queue so no additional sorting is done. You should > only have to configure connections to use the BucketPrioritizer at points in > your flow where you believe you'll have enough information to accurately > determine priority but not beyond that point unless you want to re-evaluate > downstream for some reason. There is administration involved in setting these > BucketPrioritizers on some connections but it should be minimal per flow > (sometimes as few as one). > > Some additional information: When you delete a rule the next time each > FlowFile moves that was already associated with that rule it will be > re-evaluated against the ruleset when it enters the next connection > regardless of whether or not a BucketPrioritizer was set on that connection. > Also FlowFiles that have yet to be evaluated (have yet to encounter a > BucketPrioritizer) will not be staggered. This was a design decision that if > we don't know what a priority is for a given FlowFile we should get it to > that point in the flow as soon as possible. This decision was a result of > emperical evidence that when we did stagger unevaluated data an incoming flow > of high priority data slowed its own upstream processing down once it was > identified and processed as high priority. > > Multi-tenancy: Agreed that a global priority list could be too restrictive > for multi-tenancy and should be addressed. > > Per swapping, this is an area where I admittedly need to put more thought > into my implementation because there is plenty of room for improvement. Right > now I'm just swapping files to disk in order of least to greatest priority > but they are all stored together. Therefore they're read back into memory in > order of least to greatest priority. More work should be done here. > > - Jon > > From: Joe Witt mailto:joe.w...@gmail.com>> > Sent: Thursday, October 17, 2019 8:12:52 AM > To: dev@nifi.apache.org <mailto:dev@nifi.apache.org> > Subject: EXT: Re: [Discuss] Data prioritization - A proposed solution > > Jon > > Probably some details I don't quite understand yet so responses here are to > get there... > > The concept for scheduling is interesting. Does this basically work around > the fact that we have an unfair scheduler so this has queue implementations > which pretend data is not available when it knows that there is higher > priority data available elsewhere thus returning more threads to the pool > faster to increase the likelihood that queues with higher priority data > will get served more often? > > The notion of prioritization implies there is a sorting function happen
Re: NiFi Extension Registry?
The best mechanism is the dev mailing list. The Slack workspace provides an opportunity for live conversation, but interaction is limited by timezones, and the system of record for Apache discussions is the mailing list. A thread here will allow people to contribute across time/location and document the discussion for future reference. As for beginning the code contribution journey, the best resources to start with are the Contributor Guide [1], which discusses the _process_ around contributing, and the Developer Guide [2], which discusses code concepts, behavioral patterns, testing, etc. If there is anything in here that isn’t clear, is incorrect, or needs updating, please let us know. A lot of this was written when the community was smaller and had a higher rate of experience/participation, in addition to the fact that obviously systems change over time. Improving these docs is a constant effort, and one of the places that new contributors actually provide the best feedback. We want to make the process accessible to everyone. Thanks, and looking forward to your involvement. [1] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide <https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide> [2] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 15, 2019, at 4:34 PM, Shalin Mehta wrote: > > You're very welcome, it's every bit deserved, Andy :) > > I was actually talking to a good friend and colleague about this today, so > if you wouldn't mind giving me a few more minutes of your time... We're > early into our journey, but I suspect that we'll be coming up with cases in > the near future. What's the best way to provide that feedback for NiFi > and/or the Registry, the Slack workspace? And taking it a step further, any > initial suggestions or guidance for jumping into contribution to the > codebase? I believe I saw a thread about it on the Devs list in Sept or > Aug, so I'll need to go back and look more closely, but always appreciate > any thoughts you might have. > > Thanks again for your time! > > On Tue, Oct 15, 2019 at 3:09 PM Andy LoPresto wrote: > >> Thanks for the kind words Shalin. This is something that is in development >> as Bryan mentioned. It’s a great time to contribute ideas and suggestions, >> as well as get involved with contributing code if you’re interested. We >> always welcome input from the community, especially if you have a real >> world use case for this with specific requirements and scenarios. That will >> help us build something that doesn’t just check off features, but solves >> real problems. >> >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Oct 14, 2019, at 2:24 PM, Shalin Mehta >> wrote: >>> >>> Hey Bryan, >>> >>> Thanks for the information, especially the 1.10.0 dependency. I know that >>> there's been a lot of motion toward that release in the last couple >> weeks, >>> so I'll continue keeping an eye on that while looking into the >>> functionality in NiFi Registry 0.5.0. If there are any further >> suggestions >>> for reading about the planned evolution, I'd love to know, but I'm sure >>> this will keep me busy for a while. >>> >>> Thanks again, >>> >>> Shalin >>> >>> On Mon, Oct 14, 2019 at 4:33 PM Bryan Bende wrote: >>> >>>> Hi Shalin, >>>> >>>> There is a minimal version of an extension registry as part of NiFi >>>> Registry 0.5.0. You can see an example of how to upload a NAR in the >>>> user guide [1]. >>>> >>>> The issue is that in order to use those end-points, your NAR needs to >>>> be built with NiFi 1.10.0 which hasn't been released yet. So we are >>>> still in a little bit of a waiting period, but you could always build >>>> NiFi master and set your NAR to use nifi-api 1.10.0-SNAPSHOT if you >>>> wanted to try it out. >>>> >>>> Thanks, >>>> >>>> Bryan >>>> >>>> [1] >>>> >> https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#manage-bundles >>>> >>>> On Mon, Oct 14, 2019 at 4:22 PM Shalin Mehta >>>> wrote: >>>>> >>>>> Hey all, >>>>> >>>>> I'm starting to cut my teeth on the extensibility of N
Re: NiFi Extension Registry?
Thanks for the kind words Shalin. This is something that is in development as Bryan mentioned. It’s a great time to contribute ideas and suggestions, as well as get involved with contributing code if you’re interested. We always welcome input from the community, especially if you have a real world use case for this with specific requirements and scenarios. That will help us build something that doesn’t just check off features, but solves real problems. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 14, 2019, at 2:24 PM, Shalin Mehta wrote: > > Hey Bryan, > > Thanks for the information, especially the 1.10.0 dependency. I know that > there's been a lot of motion toward that release in the last couple weeks, > so I'll continue keeping an eye on that while looking into the > functionality in NiFi Registry 0.5.0. If there are any further suggestions > for reading about the planned evolution, I'd love to know, but I'm sure > this will keep me busy for a while. > > Thanks again, > > Shalin > > On Mon, Oct 14, 2019 at 4:33 PM Bryan Bende wrote: > >> Hi Shalin, >> >> There is a minimal version of an extension registry as part of NiFi >> Registry 0.5.0. You can see an example of how to upload a NAR in the >> user guide [1]. >> >> The issue is that in order to use those end-points, your NAR needs to >> be built with NiFi 1.10.0 which hasn't been released yet. So we are >> still in a little bit of a waiting period, but you could always build >> NiFi master and set your NAR to use nifi-api 1.10.0-SNAPSHOT if you >> wanted to try it out. >> >> Thanks, >> >> Bryan >> >> [1] >> https://nifi.apache.org/docs/nifi-registry-docs/html/user-guide.html#manage-bundles >> >> On Mon, Oct 14, 2019 at 4:22 PM Shalin Mehta >> wrote: >>> >>> Hey all, >>> >>> I'm starting to cut my teeth on the extensibility of NiFi through Custom >>> Processors, and one of the big things on my mind is being able to manage >> a >>> catalog of them across an organization. >>> >>> While watching the "BYOP" presentation given by Andy at DataWorks Summit >>> Barcelona earlier this year (awesome presentation, by the way!), there >> was >>> a really quick note of a future Extension Registry within NiFi Registry. >> I >>> dug around a bit and only came up with this Confluence page outlining >> ideas >>> for such a thing: >>> >>> >> https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions >>> >>> Is there any more information out there around this effort, such as >> updated >>> notes or specific Jira(s) about it? I'd love to get a better sense of >> what >>> the community has in mind, and what the state of the addition is like >> these >>> days. Rolling my own incarnation through process and supporting tools is >>> where my head is currently sitting, but something more "permanent" and in >>> line with the direction of the Project is way more appealing. >>> >>> Thanks much in advance, >>> >>> Shalin >>
Re: Jira contributor access
Welcome. I have added you to the contributors role, Bahlul. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Oct 9, 2019, at 2:31 PM, Bahlul Haider wrote: > > Hello, > > I will like to contribute to the nifi codebase. Please add me (Jira > username: bahlulh) as a contributor. > > Thanks, > Bahlul
