NiFi-10751 - CI Tests Not Running

I made an update to a pr I was working on and it’s not running the tests again for some reason. I haven’t done any PRs since NiFi switch over to having Github run them so I might be doing something wrong. Do I need to close the PR and recreate?

RE: NiFi Losing Zookeeper Connection

the issue. I'll see if I can reproduce it and figure out why it's occurring/what we can do to fix it. I created a Jira issue here: https://issues.apache.org/jira/browse/NIFI-9560. Nathan On Mon, Jan 10, 2022 at 1:24 PM Shawn Weeks mailto:swe...@weeksconsulting.us>> wrote: In my case

RE: Primary node vs shutdown

On newer versions there is an option in the UI to Offload the data if you have NiFi's cluster load balancing setup. Then you'd disconnect the node and shut it down. Thanks Shawn -Original Message- From: Phil H Sent: Wednesday, September 1, 2021 4:36 PM To: dev@nifi.apache.org

RE: a problem about data persistence when the nifi server deployed on k8s

What are you using for your persistence layer(PV) and is NiFi using the same volume when you restart? Thanks Shawn -Original Message- From: Shigin Wang Sent: Tuesday, August 10, 2021 5:15 AM To: dev@nifi.apache.org Subject: a problem about data persistence when the nifi server

RE: PutFile to mounted directory

You could set the permissions on the unmounted folder where NiFi can't write to them since the permissions when mounted come from the NFS share. For example assume we're mounting to /mnt/nifi_dropoff, before mounting to that mount point create the directory set the permissions to 555 or

Docker Image Improvements for Kubernetes

I’m working on deploying NiFi to Kubernetes and I’ve ran across several things that could be improved. 1. Currently flow.xml.gz is stored in ./conf by default which has been designated a Docker volume. In Kubernetes volumes are not pre-populated from the image so I’m left with some init

Re: Nifi with kerberos(kdc) is not working in Nifi 1.11.4

NOKIA.COM -Original Message- From: Shawn Weeks Sent: Friday, April 17, 2020 5:43 PM To: dev@nifi.apache.org Subject: Re: Nifi with kerberos(kdc) is not working in Nifi 1.11.4 Can you verify that your KDC Realm is really NIFI.COM and that it's defined in /etc

Re: Nifi with kerberos(kdc) is not working in Nifi 1.11.4

Can you verify that your KDC Realm is really NIFI.COM and that it's defined in /etc/krb5.conf? Thanks Shawn On 4/17/20, 5:14 AM, "Ganesh, B (Nokia - IN/Bangalore)" wrote: Hi , I am facing issue with Nifi 1.11.4 in Kerberos mode , whereas nifi 1.9.2 not seeing this issue . I

Re: Nifi installation - port 8080 being used even nifi installed with a different port

You’ll need to add this line to your bootstrap.conf file to disable the Zookeeper Admin Server. We’ve done this for you in NiFi 1.11.x java.arg.17=-Dzookeeper.admin.enableServer=false From: Samarendra Sahoo Reply-To: "us...@nifi.apache.org" Date: Friday, March 6, 2020 at 11:17 PM To:

Re: Running Nifi on OpenShift

com] Sent: 13 February 2020 14:32 To: dev@nifi.apache.org; Endre Kovacs Cc: Ali, Rizwan Subject: RE: Running Nifi on OpenShift Public Hi Shawn, Thank you for your message. I will add your suggested configs and try it out today. It certainly has new content

Re: Running Nifi on OpenShift

Analyst Software Developer -Original Message- From: Shawn Weeks [mailto:swe...@weeksconsulting.us] Sent: 12 February 2020 21:16 To: dev@nifi.apache.org; Endre Kovacs Cc: Ali, Rizwan Subject: Re: Running Nifi on OpenShift I recognize that ru

Re: Running Nifi on OpenShift

ue could be due to the fact that Nifi image tries to run as root but OpenShift doesn't allow it by default. Not sure if this is still true for the latest 1.11.1 version of docker image. If you can suggest what is wrong with these yml files or may be some settings need to change on OpenShift admin

Adding Failure Information to AbstractRecordProcessor

I’d like to get some feedback on an enhancement that I’m looking at doing. I need a way for a custom Record Reader to pass information downstream on parse failures so that they can be logged for futhur investigation by a human. I’d like to have AbstractRecordProcessor add an attribute

Re: Running Nifi on OpenShift

I recognize that running NiFi on Kubernetes isn't quite as easy as starting it in Docker but it's also not that hard if you've worked with Kubernetes a bit. More than likely the issue is in your Kubernetes Yaml that you used to deploy NiFi with. This is separate than nifi.properties and would

Re: Running Nifi on OpenShift

That error is caused by incorrect permissions on the volume but I'm not sure why. Can you share your Kubernetes Yaml file, it will make it easier to track down the difference? Also what are you using for a persistence volume? Thanks Shawn On 2/12/20, 1:14 PM, "Fill, Natalia" wrote:

Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream

/10/20, 12:27 PM, "Shawn Weeks" wrote: I've been testing out NiFi on K8s quite a bit lately and might be able to help. What are you using for your persistent volume, this kinda sounds like NiFi can't always access data in the content_repository folder? Thanks Shawn

Re: change Filename

Your screen captures didn’t make it through but if your just needing to set the file name for a flow file the attribute is filename and you can change it with update attribute. Thanks Shawn Sent from my iPhone On Jan 28, 2020, at 7:11 AM, סמדג'ה גיא wrote:  Hi all Is there a way to change

Re: Any work planned for adding Kerberos pkinit support?

If you don’t mind, what is the use case where keytabs aren’t working for you? I’ve always found them easier since I don’t think you can setup passwords for service principals only user principals. Thanks Shawn From: Pat White Date: Thursday, January 23, 2020 at 4:46 PM To: Shawn Weeks Cc

Re: Any work planned for adding Kerberos pkinit support?

Spoke too soon. See https://issues.apache.org/jira/browse/NIFI-6988 On 1/23/20, 4:23 PM, "Shawn Weeks" wrote: Password based Kerberos is in the works and might have made it in 1.11. I’ve been seeing the pull requests for a while now. Thanks Shawn Sent from

Re: Any work planned for adding Kerberos pkinit support?

Password based Kerberos is in the works and might have made it in 1.11. I’ve been seeing the pull requests for a while now. Thanks Shawn Sent from my iPhone > On Jan 23, 2020, at 11:22 AM, Pat White > wrote: > > Hi Folks, > > Is there any support for Kerberos authentication using Pkinit

Re: zookeeper.admin.enableServer question

It's a typo on my part for sure but it still works that way. I've verified the parameter is added to command line on startup. Thanks Shawn On 1/22/20, 11:59 AM, "Joe Gresock" wrote: I'm checking out the 1.11.0 RC and noticed that in the packaged bootstrap.conf, the following property

Re: [VOTE] Release Apache NiFi 1.11.0 (rc3)

+1 non-binding On 1/22/20, 10:48 AM, "Shawn Weeks" wrote: Noticed that the docker compose file nifi-docker/docker-compose/docker-compose.yml still references 1.10.0-SNAPSHOT-dockermaven in RC3. Not worth holding up for though. Thanks Shawn On 1/22/2

Re: [VOTE] Release Apache NiFi 1.11.0 (rc3)

Noticed that the docker compose file nifi-docker/docker-compose/docker-compose.yml still references 1.10.0-SNAPSHOT-dockermaven in RC3. Not worth holding up for though. Thanks Shawn On 1/22/20, 9:55 AM, "Kevin Doran" wrote: +1 (binding) I followed the steps in the release

Re: Release vote on Catalina?

All of my development is on Catalina and IntelliJ and I haven’t had any trouble with the build. Thanks Shawn Sent from my iPhone > On Jan 17, 2020, at 6:25 PM, Marc Parisi wrote: > > Mike my osx builds are also Catalina based. I usually add -T2C but don't > recall if I added that to the

Re: [VOTE] Release Apache NiFi 1.11.0 (rc2)

+1 Build runs fine and hashes match. On 1/16/20, 4:54 PM, "Joe Witt" wrote: Hello, I am pleased to be calling this vote for the source release of Apache NiFi nifi-1.11.0. The source zip, including signatures, digests, etc. can be found at:

NIFI-6986 - Review

Test failures aren’t related to my changes. Please review when you have a chance. Thanks Shawn

NIFI-6881 - Review

Whenever someone has a chance can you review https://github.com/apache/nifi/pull/3896 please. I welcome comments and suggestions. Thanks Shawn

Re: [discuss] Apache NiFi 1.11.0

I've pushed a new version of NIFI-6852 with integration tests. Assuming the build completes can someone review please. Thanks Shawn On 1/12/20, 9:51 AM, "Shawn Weeks" wrote: I'm working on the test cases this morning. I'm planning on being done today if nothing comes up.

Re: [discuss] Apache NiFi 1.11.0

I'm working on the test cases this morning. I'm planning on being done today if nothing comes up. Thanks On 1/11/20, 6:09 PM, "Mark Bean" wrote: There's an open PR [1] for NIFI-6852. Could really use this in 1.11.0. [1] https://github.com/apache/nifi/pull/3973 On Mon,

Re: [discuss] Apache NiFi 1.11.0

I know there's still some stuff out on it but NIFI-6886 is broken in 1.10 and wasn't broken previously so it would be good to grab it if we can get all the issues resolved. If we don't hear back from the contributor I don't mind grabbing the pull request and adding Matt's fixes to it. Thanks

Lookup Based Schema Registry

I’m looking at building a version of the Avro schema registry that uses a lookup service. This would be for cases where you have too many schemas to easily manage from the property based avro schema registry but don’t need or want the complexities of the Confluent or Hortonworks registries.

Help With NiFi Travis Errors

I figure I’m missing something but could someone take a look at https://travis-ci.org/apache/nifi/jobs/615062300?utm_medium=notification_source=github_status and see why it can’t find the Derby Library. It’s used in another test in the same module without issue. Thanks Shawn

Re: Errors Building from Master

It was just maven being maven. Clearing .m2 and running again fixed it. On 11/20/19, 9:32 AM, "Shawn Weeks" wrote: I’m getting failures building NiFi from master on the nifi-hive3-processors like this. I’m building on a Mac with OpenJDK 8. I’m assuming I’m missing some bui

Errors Building from Master

I’m getting failures building NiFi from master on the nifi-hive3-processors like this. I’m building on a Mac with OpenJDK 8. I’m assuming I’m missing some build depency. [ERROR] Failure executing groovy-eclipse compiler: [ERROR] -- [ERROR] 1. WARNING in

Re: Revision Source for AtomicDistributedMapCacheClient

e/nifi-registry/blob/master/nifi-registry-core/nifi-registry-revision/nifi-registry-revision-spring-jdbc/src/main/java/org/apache/nifi/registry/revision/jdbc/JdbcRevisionManager.java#L133-L148 On Tue, Nov 19, 2019 at 9:01 AM Shawn Weeks wrote: > > Where does the val

Revision Source for AtomicDistributedMapCacheClient

Where does the value from revision come from on the call to replace in AtomicDistributedMapCacheClient? What should it be set to on a call to replace if a revision isn’t provided? I’m looking at the WaitNotifyProtocol class and it looks like revision isn’t even used. If that’s the case then the

Re: New Distributed Map Cache Impementations

-odbc-jdbc/ On Thu, Nov 14, 2019 at 8:31 AM Shawn Weeks wrote: > > Has anyone already done some work on adding new services for distribute map cache? I’m looking at moving to aws and I really don’t want to have to run emr just for hbase. I’ve been thinking about starting on e

New Distributed Map Cache Impementations

Has anyone already done some work on adding new services for distribute map cache? I’m looking at moving to aws and I really don’t want to have to run emr just for hbase. I’ve been thinking about starting on either a DynamoDB or simple jdbc implementation. Thanks Shawn Sent from my iPhone

NiFi Registry Docker Persistenace

It looks like the office NiFi Registry Docker Image and Dockerfile isn’t setup for persisting state in Docker Volumes. Is there an existing issue to update this or another Docker Image I can use? Thanks Shawn Weeks

Re: NiFi Toolkit CLI Token Creation

y LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 13, 2019, at 12:11 PM, Shawn Weeks wrote: > > Got it, for now I'm just going to work on implementing a Kerberos solution

Re: NiFi Toolkit CLI Token Creation

ingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jun 13, 2019, at 11:52 AM, Shawn Weeks wrote: > > > > Completely agree on username and password but I'll probably still do something somewhat generic around access tokens vs 2 way

Re: NiFi Toolkit CLI Token Creation

On Thu, Jun 13, 2019 at 2:28 PM Shawn Weeks wrote: > > Got it, I've been trying to read some of this on my phone and missed something. Currently it looks like the NiFi Client JerseyNiFiClient.java was setup to support token(JWT) based requests but from what I can tell those m

Re: NiFi Toolkit CLI Token Creation

beros auth. On Thu, Jun 13, 2019 at 1:54 PM Shawn Weeks wrote: > > Ok, I was thinking the CLI used the Rest API exclusively and that's what I was missing. Unfortunately I don't have the option to use self-signed certificate due to organizational security policies and we don't ha

Re: NiFi Toolkit CLI Token Creation

o.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > On Jun 13, 2019, at 10:27 AM, Shawn Weeks wrote: > > > > Is there a way to pass 2 currently? Because you can get the token via curl like I’m currently doing? > &

Re: NiFi Toolkit CLI Token Creation

n of the authentication aspects of the CLI > toolkit. > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 13, 2019, at 10:00 AM, Shawn Weeks wrote: >&

Re: NiFi Toolkit CLI Token Creation

ex.html> >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> > On Jun 12, 2019, at 9:39 AM, Shawn Weeks >> wrote

NiFi Toolkit CLI Token Creation

I work in an environment reluctant to create self signed ssl certificates and I’m looking at the feasibility of having the toolkit cli authenticate via Kerberos. I was expecting it to be as simple as adding another way to get the authentication token but I’m having trouble figuring out exactly

Re: Get IOException occasionally

There are several complications using Snappy but from what I can tell the SelectHiveQL Processor isn't trying to use it and instead the Hive JDBC Driver is trying to use it. Are you running a specific vendors distribution of NiFi? The reason I'm asking is when I go look at the source for the

RE: Adding HBase Support for AtomicDistributedMapCacheClient

Another item that seems to fail is the call to DiskUtils.deleteRecursively(rootFile); in TestFileSystemRepository. Is there a reason why rootFile.delete() isn't sufficient? It deletes directories as well. Thanks Shawn Weeks -Original Message- From: Shawn Weeks Sent: Saturday, May 4

RE: Adding HBase Support for AtomicDistributedMapCacheClient

T_RES_PATH = NiFiGroovyTest.getClassLoader().getResource(".").toURI().getPath() See posts like https://stackoverflow.com/questions/43972777/exception-in-thread-main-java-nio-file-invalidpathexception-illegal-char for an explanation. Thanks Shawn Weeks -Original Message----- From: Shawn We

RE: Adding HBase Support for AtomicDistributedMapCacheClient

and I'll implement it. Thanks Shawn Weeks -Original Message- From: Bryan Bende Sent: Thursday, April 25, 2019 7:05 PM To: dev@nifi.apache.org Subject: Re: Adding HBase Support for AtomicDistributedMapCacheClient Should be available through the existing scan methods, they take a ResultHandler

RE: Adding HBase Support for AtomicDistributedMapCacheClient

legate back to the > original checkAndPut and ignore the timestamp, and document that it > isn't used in that implementation. I don't love this, but it does > allow both services to implement the functionality and still leverage > the better solution for 2_x. > > > On Thu, Apr 2

RE: Adding HBase Support for AtomicDistributedMapCacheClient

it needs to call? Just so I can get an idea of the differences between 1.x and 2.x. On Thu, Apr 25, 2019 at 1:00 PM Shawn Weeks wrote: > > While checkAndPut is atomic as it's built now it doesn't support also > checking the timestamp range which is included in the new checkAndMutate API.

RE: Adding HBase Support for AtomicDistributedMapCacheClient

://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-redis-bundle/nifi-redis-extensions/src/main/java/org/apache/nifi/redis/service/RedisDistributedMapCacheClientService.java#L271 On Thu, Apr 25, 2019 at 12:38 PM Shawn Weeks wrote: > > I'll need to add a check and mutate

RE: Adding HBase Support for AtomicDistributedMapCacheClient

for AtomicDistributedMapCacheClient I'm not aware of a JIRA, so I'd say go for it. On Wed, Apr 24, 2019 at 9:27 PM Shawn Weeks wrote: > > Seems like this should be fairly easy for HBase 2.x with the checkAndMutate > functionality and I was wondering if there is already a Jira for this. &g

Adding HBase Support for AtomicDistributedMapCacheClient

Seems like this should be fairly easy for HBase 2.x with the checkAndMutate functionality and I was wondering if there is already a Jira for this. Otherwise I might make an attempt at it. It would be good to be able to support Wait/Notify and other things that need