Re: Is there a need for the ConvertAvroToJson processor in 2.x?

2024-04-10 Thread Mike Thomsen
IIRC, the plan was to remove those ConvertToX processors in 2.X On Wed, Apr 10, 2024 at 11:36 AM Dan S wrote: > I noticed that the ConvertAvroToJson processor is not marked as deprecated > in 2.x. Is there something it offers which cannot be done with the > ConvertRecord processor configured

Re: [discuss] What to do with the Cassandra components

2024-03-22 Thread Mike Thomsen
ctive mx > especially as it relates to vulns. > > Thanks > Joe > > On Fri, Mar 22, 2024 at 11:42 AM Mike Thomsen > wrote: > > > The scope tag was probably copy pasta. You raise a valid point about the > > processor dependencies that completely slipped my mind. That s

Re: [discuss] What to do with the Cassandra components

2024-03-22 Thread Mike Thomsen
use it conflicts with the ScyllaDB > > java-driver-core in the implementation NAR. > > > > This is the reason Matt and I highlighted for providing a layer of > > abstraction at the Controller Service API level. > > > > Regards, > > David Handermann > > >

Re: [discuss] What to do with the Cassandra components

2024-03-22 Thread Mike Thomsen
Work so far: https://github.com/MikeThomsen/nifi/tree/cql-changes On Thu, Mar 21, 2024 at 9:52 AM Mike Thomsen wrote: > Matt/David, > > By this evening, I should be at a point where I can share my branch. It > should be far enough along that y'all can see what I mean ab

Re: [discuss] What to do with the Cassandra components

2024-03-21 Thread Mike Thomsen
/parameters (I envision an option where the user puts EL in the file, we load the file, preprocess the EL and load that into the driver) On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen wrote: > If it were that simple, they would probably have just gone with that > solution. That said, t

Re: [discuss] What to do with the Cassandra components

2024-03-20 Thread Mike Thomsen
wever I'm still leaning towards a vendor-agnostic API. > > On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen > wrote: > > > At first glance, the package names look identical to me: > > > > https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html >

Re: [discuss] What to do with the Cassandra components

2024-03-20 Thread Mike Thomsen
abstraction in NiFi that avoids depending on either library at the > Controller Service API level. > > Regards, > David Handermann > > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen > wrote: > > > > Matt/David, > > > > I got some time this morning to take a

Re: [discuss] What to do with the Cassandra components

2024-03-20 Thread Mike Thomsen
it likely now (due to the refactor) that we will simply be able to > > upgrade the driver when Cassandra 5 is GA? Also does anyone use Netflix's > > Astyanax [1]? > > > > [1] > > > https://cassandra.apache.org/doc/stable/cassandra/getting_started/driv

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
y be able to > upgrade the driver when Cassandra 5 is GA? Also does anyone use Netflix's > Astyanax [1]? > > [1] > https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java > >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen wrote: >> >> Realist

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
> the kind of abstraction that would be ideal, and I believe that also > aligns with what Matt has described. > > Regards, > David Handermann > > On Tue, Mar 19, 2024 at 1:45 PM Mike Thomsen > wrote: > > > > ** we can dump v3 **DRIVER** compatibility, since later

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
** we can dump v3 **DRIVER** compatibility, since later 4.X Java drivers are backward compatible with Cassandra 3 On Tue, Mar 19, 2024 at 2:43 PM Mike Thomsen wrote: > David, > > Before we proceed, I think we should make sure we're all understanding the > same problem he

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
d Handermann > > [1] https://github.com/apache/cassandra-java-driver/ > > On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen > wrote: > > > > Matt, > > > > I got that. My point was that the Java changes appear to be a one time > > thing that DataStax did t

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
tainable code in the future by not having to change any > > processors, just the controller service implementation. > > > > > > On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen > > wrote: > > > > > > > > > > > https:/

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
024 at 10:13 AM Mike Thomsen wrote: > Matt, > > I don't think we need to really "abstract above" the drivers because the > Java DataStax driver appears to support 4.X all the way back to 2.X, as > well as the enterprise versions from DataStax > > https://docs.datas

Re: [discuss] What to do with the Cassandra components

2024-03-19 Thread Mike Thomsen
> current set of components? > > > > > >> > >> > > > > > >> > >> The CVE hits are concerning and long standing. Supporting > > > > > Cassandra > > > > > >> 3 > > > > > >> > >> im

Re: [discuss] What to do with the Cassandra components

2024-03-15 Thread Mike Thomsen
That’s been on my todo list for a little while but things kept coming up. I think I could get started on that now. Based on my initial research it appears that scylla uses the exact same api as datastax so supporting both in a cql bundle should theoretically be fairly easy. Sent from my

Re: [DISCUSS] New schema repository idea (with proof of concept)

2024-03-07 Thread Mike Thomsen
JAR for referencing in this > proposed registry? > > Regards, > David Handermann > > On Thu, Mar 7, 2024 at 4:30 PM Mike Thomsen > wrote: > > > > You raise some good points, but I think there's still ample room for > > file-based schema registries within NiFi.

Re: [DISCUSS] New schema repository idea (with proof of concept)

2024-03-07 Thread Mike Thomsen
t in NiFi itself. > > Those are my initial thoughts, perhaps others can provide additional > perspective. > > Regards, > David Handermann > > On Sat, Mar 2, 2024 at 9:18 AM Mike Thomsen > wrote: > > > > I've had this project on the back burner for a while an

[DISCUSS] New schema repository idea (with proof of concept)

2024-03-02 Thread Mike Thomsen
I've had this project on the back burner for a while and wanted to share it with the team. It's a schema repository implementation that is designed to take a JAR file with POJOs and use Jackson's schema generation API to generate Avro schemas from those on startup. It also uses (via Jackson) Avro

Re: UpdateAttribute Failure Relationship

2024-02-09 Thread Mike Thomsen
How about a third option which is to provide three options: 1) Default - status quo, exceptions cause it to yield 2) Exception = moves forward to success w/ an error attribute, an error log statement that triggers a bulletin, etc to let data manages know what's happening. 3) Exception = moves to

Re: UpdateAttribute Failure Relationship

2024-02-09 Thread Mike Thomsen
Obviously, three options in a drop down :-D On Fri, Feb 9, 2024 at 7:49 AM Mike Thomsen wrote: > How about a third option which is to provide three options: > > 1) Default - status quo, exceptions cause it to yield > 2) Exception = moves forward to success w/ an error attribute, an

Re: How to get files after a condition has been met

2024-02-06 Thread Mike Thomsen
I think the easiest way to do the conditional fetch would be to use ConsumeKafka (or something equivalent for this use case like ListenHttp) and FetchFile. On Mon, Feb 5, 2024 at 4:23 PM Dan S wrote: > I have a requirement to load Json Schemas into the > StandardJsonSchemaRegistry from the file

Re: Removing JRuby?

2023-11-07 Thread Mike Thomsen
n the Python stuff > > before we'd want to deprecate the scripted bundle. > > > > On the JRuby front, is that something you use actively? This question > > is for you and the entire community of course. > > > > Regards, > > Matt > > > > On Mon,

Re: Removing JRuby?

2023-11-06 Thread Mike Thomsen
g forward to everyone's thoughts! > > - Matt > > On Sun, Nov 5, 2023 at 8:31 PM Mike Thomsen > wrote: > > > > https://issues.apache.org/jira/browse/NIFI-11646 > > > > I get the removal of Lua, but not the removal of JRuby. It's a clean > > reimple

Removing JRuby?

2023-11-05 Thread Mike Thomsen
https://issues.apache.org/jira/browse/NIFI-11646 I get the removal of Lua, but not the removal of JRuby. It's a clean reimplementation of Ruby native to the JVM and AFAICT is pound for pound as actively maintained as Groovy. Also, at this point, does it make sense to even keep the groovyx bundle

Default parameter storage

2023-11-02 Thread Mike Thomsen
I'm trying to track down the parameter sync issue I posted earlier. My understanding is that the parameters are stored in the Registry unless you configure your own parameter provider. Is that right? We're just using a Registry instance without any custom providers. Trying to figure out why we

Re: discuss NiFi 1.24 release

2023-10-12 Thread Mike Thomsen
gt; handled automated migration from H2 1.4 to 2.1, and 2.1 to 2.2, so the > basic mechanics are in place to extract content from H2. The general > upgrade path will be to migrate to the latest version 1 release, and > then upgrade to version 2. > > Regards, > David Handermann &g

Re: discuss NiFi 1.24 release

2023-10-12 Thread Mike Thomsen
When H2 goes, what will the upgrade path look like? On Tue, Oct 10, 2023 at 7:57 PM David Handermann < exceptionfact...@apache.org> wrote: > Joe, > > Thanks for the reply, that sounds good. > > For reference, here is the Jira issue for tracking the initial > implementation: > >

Re: [discuss] nifi 2.0 and Java 21…

2023-09-06 Thread Mike Thomsen
+1 100% On Wed, Sep 6, 2023 at 11:48 AM Adam Taft wrote: > Yes, please. +1 Exactly what Mark said. Virtual threads have potential to > be extremely impactful to applications like NiFi. > > /Adam > > On Wed, Sep 6, 2023 at 7:26 AM Mark Payne wrote: > > > Thanks for bringing his up, Joe. > > > >

Validation on white spaces in record api

2023-08-28 Thread Mike Thomsen
I lost a lot of time this morning debugging an issue that turned out to just be a trailing whitespace char at the end of an alias value. That seems like a good place to put extra validation in the record api; I can't imagine a legit use case where someone would intentionally put trailing or

Re: Consultation on Learning and Development in Apache NiFi

2023-08-22 Thread Mike Thomsen
When you say create connectors, do you mean create custom NiFi components? On Thu, Aug 17, 2023 at 11:46 AM Frank Mansilla wrote: > Dear members of the Apache NiFi community, > > I hope this message finds you well. My name is Frank Mansilla and I am a > data analyst based in Bolívar, Buenos

Re: Processing JSON Files Embedded with Base64 Encoded Files

2023-08-15 Thread Mike Thomsen
Can you be more specific on what you're trying to accomplish since the RecordPath API has base64 encode/decode functions for stuff like decoding inline base64 content? On Tue, Aug 15, 2023 at 9:45 AM Reid, Chris wrote: > Hello. > > I understand that there's currently no Apache NiFi processor

Re: Java 17 features in 2.x

2023-08-11 Thread Mike Thomsen
; > > > We already have a few Java 11 and 17 references on the main branch > for > > > > things like List.of(), and most of these are easy to adjust when > > > > backporting, but they do require careful attention. > > > > > > > > Rega

Java 17 features in 2.x

2023-08-07 Thread Mike Thomsen
Since we're standardizing on 17, we're free and clear to use Java 17 features, right?

H2 CVE

2023-06-22 Thread Mike Thomsen
A colleague found this "CVE" report for H2. I agree with the H2 devs that it's a big joke of a CVE, but it's something we might want to add something to the documentation to discuss because it could cause grief for our users. https://github.com/h2database/h2database/issues/3686

Re: Possible Docker bug in 1.13.2

2023-06-07 Thread Mike Thomsen
> Because I have custom processors that cannot be rebuilt (source-code gone) IntelliJ can probably give you a close version of the source code if you expand the JAR and drag the class files for your custom code into the IDE. On Wed, Jun 7, 2023 at 5:32 PM Russell Bateman wrote: > I downloaded

Re: [discuss] nifi 2.0 and Java 17...

2023-05-31 Thread Mike Thomsen
I'm all in favor of making 17 the requirement for 2.0 On a related note, I was asked by one of our devops guys if 17 was a good idea for 1.X. I know we support it, but is it considered just as good as 11 for 1.21+? On Wed, May 31, 2023 at 6:15 PM Kevin Doran wrote: > Yeah I agree it makes

Re: Help in creating a custom record reader

2023-05-15 Thread Mike Thomsen
the implementation details for everything from the Record components to the core reader implementations. On Mon, May 15, 2023 at 8:54 AM Mike Thomsen wrote: > You would need to create a new Maven project that has a root POM and two > modules (one for the JAR and one for the NAR). You can find ex

Re: Help in creating a custom record reader

2023-05-15 Thread Mike Thomsen
You would need to create a new Maven project that has a root POM and two modules (one for the JAR and one for the NAR). You can find examples of that project structure in our repository or in a lot of community repos on GitHub. >From there, you'd implement (going on memory here without the

Write access to Confluence

2023-05-04 Thread Mike Thomsen
Could someone with permissions to do it, grant me (mthomsen) write access to our Confluence pages? Thanks, Mike

Deprecating the use of older Java date formatting code

2023-04-20 Thread Mike Thomsen
A new contributor raised some good questions on a PR: https://github.com/apache/nifi/pull/7169 I rebased this branch as we discussed in the original PR, but I see that meantime NiFi code become much more dependent on DateFormat. I see many public methods in DataTypeUtils taking or returning

Re: Are all 1.21 NARs uploaded as Maven artifacts?

2023-04-10 Thread Mike Thomsen
By it, I mean this: https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/nifi-azure-nar-1.21.0.nar On Mon, Apr 10, 2023 at 1:07 PM Mike Thomsen wrote: > When I clicked on it, I got a 404. Weird. > > On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende wrote: > >

Re: Are all 1.21 NARs uploaded as Maven artifacts?

2023-04-10 Thread Mike Thomsen
When I clicked on it, I got a 404. Weird. On Mon, Apr 10, 2023 at 12:53 PM Bryan Bende wrote: > The Azure NAR looks like it is there: > > https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-azure-nar/1.21.0/ > > On Mon, Apr 10, 2023 at 12:40 PM Mike Thomsen > wrote: &g

Re: Are all 1.21 NARs uploaded as Maven artifacts?

2023-04-10 Thread Mike Thomsen
ed and uploaded. It might be > that we're not pushing Atlas nars anymore? > > I dont recall but that is possible. > > Thanks > >> On Mon, Apr 10, 2023 at 9:21 AM Mike Thomsen wrote: >> >> Getting this error on two different machines on two different ne

Are all 1.21 NARs uploaded as Maven artifacts?

2023-04-10 Thread Mike Thomsen
Getting this error on two different machines on two different networks: ownloaded from central: https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-sql-reporting-nar/1.21.0/nifi-sql-reporting-nar-1.21.0.nar (26 MB at 183 kB/s) Downloaded from central:

Re: [VOTE] Release Apache NiFi 1.21.0 (RC2)

2023-04-05 Thread Mike Thomsen
+1 binding. On Wed, Apr 5, 2023 at 5:53 AM Csaba Bejan wrote: > +1 (non-binding) > > - Went through the helper guide and did a clean build > - Verified signatures and hashes > - Built on OSX 11.6.6 > - AdoptOpenJDK (build 1.8.0_282-b08) > - Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)

Re: [VOTE] Release Apache NiFi 1.21.0 (RC1)

2023-03-30 Thread Mike Thomsen
+1 Binding On Wed, Mar 29, 2023 at 6:01 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache > NiFi 1.21.0. > > The source zip, including signatures, digests, etc. can be found at: >

Re: 1.X branch

2023-02-21 Thread Mike Thomsen
> 'support/nifi-1.x' line is the correct one though for what is > currently 1.21.0-SNAPSHOT. > > Just please be sure nothing goes there that doesn't also go to main > (2.0.0-SNAPSHOT) unless it is a bug fix for something only present on > 1.x line. > > Thanks > Joe > &

1.X branch

2023-02-21 Thread Mike Thomsen
Is apache/support/nifi-1.x the correct branch to merge 1.21+ changes to? Thanks, Mike

Re: ValidateCsv vs ValidateRecord

2023-01-28 Thread Mike Thomsen
Also, I think there will be a high likelihood of format-specific processors like ValidateCSV going away in NiFi 2.0. I wouldn't start any new data engineering work using these processors. On Wed, Jan 18, 2023 at 4:02 PM Pierre Villard wrote: > > Hi, > > ValidateCSV was released before and is

Re: [VOTE] Adopt NiFi 2.0 Proposed Release Goals

2022-12-13 Thread Mike Thomsen
+1 binding On Tue, Dec 13, 2022 at 9:33 AM Mark Bean wrote: > > +1 (non-binding) > > On Mon, Dec 12, 2022 at 12:02 PM David Handermann < > exceptionfact...@apache.org> wrote: > > > Team, > > > > Following positive feedback on NiFi 2.0 Proposed Release Goals [1] on the > > recent discussion

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

2022-12-07 Thread Mike Thomsen
a 17 needs to be identified and fixed as well as > more thorough testing to find other possible edge cases before we move > forward too aggressively. > > [1] https://issues.apache.org/jira/browse/NIFI-10958 > > On Wed, Dec 7, 2022 at 1:33 PM Mike Thomsen wrote: > > > Really

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

2022-12-07 Thread Mike Thomsen
Really good start on the discussion. One thing I'm curious about is Java 11 vs 17. Java 8 -> 11 is major jump that I can understand why businesses scoffed at that for a long time, but the lift from 11 to 17 was about like 7 -> 8. A 2.0 release seems like a good time to jump straight to the latest

Optional build profiles with Category X licenses

2022-11-27 Thread Mike Thomsen
I know the official binaries cannot contain Category X license code, but is it allowable to have optional build profiles that integrate Category X licensed code? An example scenario that lead me to wonder if this is doable is I would like to try to add optional support for Elastic's Painless as

Re: merge PR 6335

2022-11-21 Thread Mike Thomsen
"doodzio," as far as I can tell, is not a committer or PMC member so it's not been approved by someone with authority to finish a review. On Sun, Nov 20, 2022 at 11:59 PM Akshara Uke wrote: > > Hello, > > Requesting merge of https://github.com/apache/nifi/pull/6335 , the review > is already

Re: [discuss] NiFi 1.19.0

2022-11-18 Thread Mike Thomsen
Can someone do a quick review on this? I think it would be a nice little bullet point on our release to say we added preliminary support for Neo4J 5 since it's so new. https://github.com/apache/nifi/pull/6684 (Enabling the V5 profile has a hard dependency on Java 17) On Fri, Nov 18, 2022 at

Re: Adding Kotlin support to the scripting bundle

2022-11-07 Thread Mike Thomsen
bilities. I hesitate to suggest this for any of the > > other scripting languages but as Kotlin becomes more popular and given > > its tight integration with the JVM (vs Jython for example), that might > > be the way to go. > > > > Regards, > > Matt

Re: Import Talend's .jar to Nifi

2022-11-07 Thread Mike Thomsen
If you mean "can you derive a NiFi flow automatically from a Talend jar" then no. With any jar, if the jar is a fat jar that doesn't rely on an external framework (ex Spark, Flink or MapReduce) then you can conceivably load the logic with ExecuteScript using the jar as a module. On Mon, Nov 7,

Adding Kotlin support to the scripting bundle

2022-11-06 Thread Mike Thomsen
I finally chased down the behavior differences that I found and discussed with Matt in the previous PR. Kotlin's got very promising (long term) JSR223 support but with a few caveats in the short term: 1. Its ScriptEngine does not support ScriptEngine#get properly, so any component that uses

Re: How to download CSV from web or git into Nifi server

2022-10-31 Thread Mike Thomsen
I think InvokeHttp would meet your needs. On Sun, Oct 30, 2022 at 1:33 AM Anand Bheemireddy wrote: > > Any response > > On Thu, Oct 27, 2022 at 2:12 PM Anand Bheemireddy < > bheemireddyan...@gmail.com> wrote: > > > Hi team, > > > > I got a requirement to download files(CSV) from web and save

Re: Need to deprecate the deduplication functionality in the MongoDB GridFS processors

2022-10-28 Thread Mike Thomsen
NiFi, and it seems like having a transitional way forward would > be helpful. If the Testcontainers change can verify the current MD5 > functionality, that should provide a good baseline for a subsequent PR to > implement a new hashing strategy. > > Regards, > David Handermann &

Re: Need to deprecate the deduplication functionality in the MongoDB GridFS processors

2022-10-25 Thread Mike Thomsen
. That's why I'm not sure we should be dogmatic about waiting. On Tue, Oct 25, 2022 at 2:15 PM Pierre Villard wrote: > > IMO we should start working on NiFi 2.0 going forward and it sounds like a > good opportunity to make such changes in our components. > > > Le mar. 25 oct.

Need to deprecate the deduplication functionality in the MongoDB GridFS processors

2022-10-25 Thread Mike Thomsen
The hash-based deduplication strategy used the built-in "md5" attribute to offload the work to the database. That functionality was deprecated and AFAICT gone as of Mongo 5: https://www.mongodb.com/docs/manual/core/gridfs/#files.md5 I am proposing two changes: * Remove deduplication * Create a

Re: Suggestion for a new Expression Language method

2022-10-14 Thread Mike Thomsen
t; > but > I am not sure what category of functions this would fall under. > I was thinking perhaps under Boolean Logic or Searching. Does that make > sense or is another category necessary? > > On Fri, Oct 7, 2022 at 4:12 PM Mike Thomsen wrote: > > > NP. I'd recommen

Re: SSLContext from PEM files

2022-10-12 Thread Mike Thomsen
background > would help evaluate the best approach. > > Regards, > David Handermann > > On Tue, Oct 11, 2022 at 12:36 PM Mike Thomsen > wrote: > > > Neo4J for some reason doesn't support the standard Java keystore types > > or P12 files for its client SSL con

SSLContext from PEM files

2022-10-11 Thread Mike Thomsen
Neo4J for some reason doesn't support the standard Java keystore types or P12 files for its client SSL configuration. It requires the use of PEM files. Would it be better to extend the SSLContext service types to include support for PEM files or create an all new SSL Provider type that is geared

Re: Suggestion for a new Expression Language method

2022-10-07 Thread Mike Thomsen
re > > > > > > > We dont have guaranteed thread safety until the jackson 3.0 release. > > > > > > > > > Has that changed? > > > > > > On Fri, Oct 7, 2022 at 11:55 AM Mike Thomsen > > > wrote: > > > > > > &

Re: Suggestion for a new Expression Language method

2022-10-07 Thread Mike Thomsen
To do it right, you want to use a tool like Jackson or Gson to do a test parse after you've detected open and close statements that seem to be JSON open and close statements. On Fri, Oct 7, 2022 at 11:38 AM Dan S wrote: > > Mike, > What do you mean by "this might be a somewhat heavy method so

Re: Suggestion for a new Expression Language method

2022-10-07 Thread Mike Thomsen
I think this would be a fairly easy lift because Jackson's ObjectMapper in 2.X is thread safe. We'd just need to put a warning in the documentation saying "this might be a somewhat heavy method so be careful." I can make a PR for this pretty quickly. On Fri, Oct 7, 2022 at 11:11 AM David

Re: Queries regarding Nifi custom processor for Azure Data Explorer

2022-10-03 Thread Mike Thomsen
I think that is at least partly doable within NiFi (ex yes, you can restrict processors to the primary node in a cluster), but I would recommend you consider a different approach for NiFi. Unlike Spark, NiFi is perfectly content to stream in huge amounts of data continuously without any temporary

Re: Timeouts on Windows builds

2022-09-29 Thread Mike Thomsen
s real and will always be... > We just need to keep improving it as things change in those > environments. > > On Thu, Sep 29, 2022 at 8:26 AM Mike Thomsen wrote: > > > > Noticing a lot of timeouts on the Windows builds I've been restarting > > lately. We don't consider thos

Timeouts on Windows builds

2022-09-29 Thread Mike Thomsen
Noticing a lot of timeouts on the Windows builds I've been restarting lately. We don't consider those blockers if the Linux/macOS builds succeed, right?

Re: Testcontainers for managing integration tests

2022-09-28 Thread Mike Thomsen
egistry-test > > Pierre > > Le mer. 28 sept. 2022 à 19:22, Mike Thomsen a > écrit : > > > We've been successfully using testcontainers (the FOSS project) where > > I work for Elastic, Cassandra and Postgres. Was wondering if anyone > > had any opinions about starting t

Testcontainers for managing integration tests

2022-09-28 Thread Mike Thomsen
We've been successfully using testcontainers (the FOSS project) where I work for Elastic, Cassandra and Postgres. Was wondering if anyone had any opinions about starting to integrate this project where we can (again, IT only) URL for those interested: https://www.testcontainers.org/ Not sure if

Re: Nifi 1.17 does exit after start

2022-09-25 Thread Mike Thomsen
Probably didn't apt install openjdk on the machine. It's not installed by default on Ubuntu. On Wed, Sep 21, 2022 at 10:44 AM Joe Witt wrote: > > Hello > > You will want to check the /logs/nifi-app.log for clues on what > happened. > > Thanks > > On Wed, Sep 21, 2022 at 7:43 AM Froim Teschner >

Re: Nifi python usage

2022-09-15 Thread Mike Thomsen
In general, I would recommend moving the Python code to Flask-based rest services running on something like Kubernetes. That way you can access them with InvokeHttp which has a much lower overhead than ExecuteStreamCommand (no CPython startup per run). Creating containerized Python services is not

Re: H12 database lock issue in NIFI

2022-08-23 Thread Mike Thomsen
> is there any professional services help provided for nifi ? Cloudera provides great support (I'm not a Cloudera employee). They would be a good option. Another option would be to try updating to 1.16.X or 1.17. I think somewhere in there, the H2 version was updated so you might get a better

Re: [discuss] nifi 1.17

2022-07-23 Thread Mike Thomsen
I would put this one on the list of "potential blockers" for 1.17: https://github.com/apache/nifi/pull/6226 On Fri, Jul 22, 2022 at 4:52 PM Joe Witt wrote: > > Hello Everyone, > > Somehow Apache NiFi 1.16 is already four months old and we've landed just > under 300 JIRAs [1] on the 1.17 line.

Re: How to manage security artifacts from a custom processor

2022-07-06 Thread Mike Thomsen
> However, where/how do I install the certificates I'll be given for use? The JDK keytool can be used to create the JKS artifacts you need. On Tue, Jul 5, 2022 at 11:33 PM Russell Bateman wrote: > > I appreciate the responses. I will try out the canonical > /StandardSSLContextService/ first

Re: How to manage security artifacts from a custom processor

2022-07-05 Thread Mike Thomsen
Example of how to do this: https://github.com/apache/nifi/blob/aa61494fc3a68b4806784f67ad837ee821d26da4/nifi-nar-bundles/nifi-standard-services/nifi-oauth2-provider-bundle/nifi-oauth2-provider-service/src/main/java/org/apache/nifi/oauth2/OAuth2TokenProviderImpl.java On Tue, Jul 5, 2022 at 8:31

Re: FW: Apache NIFI service SRG

2022-06-24 Thread Mike Thomsen
Patrick, I sent you an email off the mailing list that might help you get what you're looking for. I assume the right email for you is your email listed here minus ".invalid." If not, sync up with me off list and I think my company has some folks who can help you out. Thanks, Mike On Fri, Jun

Re: Upgrading Cassandra driver and bringing in breaking changes with it

2022-06-14 Thread Mike Thomsen
lementation in the > > Jira issue would be helpful. > > > > Regards, > > David Handermann > > > > On Fri, Jun 10, 2022 at 3:06 PM Mike Thomsen > > wrote: > > > > > https://issues.apache.org/jira/browse/NIFI-9770 > > > >

Upgrading Cassandra driver and bringing in breaking changes with it

2022-06-10 Thread Mike Thomsen
https://issues.apache.org/jira/browse/NIFI-9770 Based on David's description saying it will require some deep changes to how various properties are configured, I think this ticket would be a good time to simplify the Cassandra bundle in general to just use a controller service to configure the

Re: Regarding Creating Email Alerts and Monitoring Tasks

2022-06-10 Thread Mike Thomsen
Instead of creating an api/webhooks, you can just use a message queue like Kafka, ActiveMQ, etc. Have NiFi subscribe to a queue and act on inputs it gets from other systems. Does that answer your question? On Thu, Jun 9, 2022 at 1:38 PM Shreyas Srinivasa wrote: > > Hi, > > I am Shreyas working

Re: [discuss] nifi 1.16.2

2022-05-10 Thread Mike Thomsen
Joe/Mark, Correct me if I'm wrong, but 9993 is not an issue if you just turn off content archiving, right? Thanks, Mike On Mon, May 9, 2022 at 12:49 PM David Handermann wrote: > > Joe, > > Thanks for initiating this discussion and moving toward 1.16.2. > > The following two issues should be

Re: [VOTE] Release Apache NiFi 1.16.1

2022-04-27 Thread Mike Thomsen
+1 (binding) Ran a simple tika extraction flow on PDFs and DOCX files. Worked fine. Saved that flow into the NiFi Registry. On Tue, Apr 26, 2022 at 4:47 PM Matt Burgess wrote: > > +1 (binding) > > Ran through release helper, tested various flows including version > control with the Registry,

Anyone else seeing this on macOS with the RC1 build?

2022-04-25 Thread Mike Thomsen
This laptop can be a little wonky at times because of the corporate lockdowns, so I wanted to see if any other committers/PMC members have seen this in nifi-record-path: [INFO] Running org.apache.nifi.record.path.util.TestFieldValueWalker [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0,

Re: Missing ECMAScript option with ExecuteScript Processor

2022-04-23 Thread Mike Thomsen
The Nashorn Javascript engine used to be bundled with the JRE, but it was removed with JRE 15. https://en.wikipedia.org/wiki/Nashorn_(JavaScript_engine) On Sat, Apr 23, 2022 at 6:02 PM Bryan Doherty wrote: > > Hi, > > When I add an ExecuteScript block to the canvas, ECMAScript is NOT a Script

1.16.1 RC?

2022-04-11 Thread Mike Thomsen
Do we have any particular tasks left under review that are blocking the 1.16.1 RC?

Re: Compiling / Testing a single processor

2022-04-09 Thread Mike Thomsen
mvn test -Dtest= from the module where the test resides. On Sat, Apr 9, 2022 at 7:21 AM Phil H wrote: > > Hi there, > > Now that my environment is working nicely, I am working on SplitContent > (Nifi-1517). > > Is there a way to tell maven to only compile and/or run the unit tests for > the

Re: Need some feedback on how upgrading Avro might cause problems

2022-04-07 Thread Mike Thomsen
I noticed because I've > > > been > > > > using that to copy-paste schemas that were "almost right" so I could > > > > manually fix them. > > > > > > > > I guess that inferred schemas should be fine if the inferring logic is > > > > als

Deprecate Livy support?

2022-04-07 Thread Mike Thomsen
https://github.com/apache/incubator-livy Looks like Livy hasn't had any updates in about two years. Maybe it's time to deprecate for it since it looks like it hasn't made any progress in moving out of the incubator.

Need some feedback on how upgrading Avro might cause problems

2022-04-07 Thread Mike Thomsen
Thread is here for full details: https://github.com/apache/nifi/pull/5900#pullrequestreview-922490039 It looks like Avro 1.8's schema parser may have been more lenient (or buggy) in enforcing the specification with respect to the ordering of a union for a nullable type. 1.9.X and higher are

Re: New build problem, Java 17

2022-04-06 Thread Mike Thomsen
there. Commercial and government setups are often filled with security stuff that messes with Maven. On Wed, Apr 6, 2022 at 8:02 PM Mike Thomsen wrote: > > > However it’s not a network issue because… > > Could be something funky with how Java 17's TLS APIs are working on your > setup. >

Re: New build problem, Java 17

2022-04-06 Thread Mike Thomsen
> However it’s not a network issue because… Could be something funky with how Java 17's TLS APIs are working on your setup. Java 8 and Java 11 are your best bets for now anyway. Java 17 support is still in the early stages and isn't a preferred JDK/JRE. On Wed, Apr 6, 2022 at 7:00 PM Phil H

Re: [discuss] nifi 1.16.1

2022-04-04 Thread Mike Thomsen
I have a few PRs for standardizing and updating our dependencies like Jackson and Avro. Might be good to get those included in 1.16.1 On Mon, Apr 4, 2022 at 3:02 PM Joe Witt wrote: > > Team, > > Sorry for the delays in wrapping up the 1.16 release from earlier last > week. We had great vote

Re: Missing maven dependencies when building nifi

2022-04-04 Thread Mike Thomsen
Did you run "mvn install" from the root of the project before attempting to run a full test of the code base? On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: > > Hi there, > > So I got past this issue by running "mvn test" from the root of the nifi > tree. After much compiling and testing, I am

Re: [ANNOUNCE] New Apache NiFi Committer Adam Markovics

2022-03-28 Thread Mike Thomsen
Congrats, Adam! On Mon, Mar 28, 2022 at 10:15 AM Pierre Villard wrote: > > Congrats Adam! > > Le lun. 28 mars 2022 à 16:11, Kevin Doran a écrit : > > > Congrats, Adam! > > > > On Mar 28, 2022 at 10:08:24, Otto Fowler wrote: > > > > > Congratulations! > > > > > > From: Marton Szasz > > >

Stacktrace from upgrading to Jackson 2.13.X

2022-03-25 Thread Mike Thomsen
I hit a snag updating the codebase to Jackson 2.13.X. Stacktrace looks familiar, like one of the Java 9/11 refactorings that moved code out of the RT into third party modules. Any ideas? [ERROR]

Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)

2022-03-23 Thread Mike Thomsen
+1 binding Ran the full build on Java 17 and tested DeduplicateRecord with a Cassandra DMC. On Wed, Mar 23, 2022 at 7:45 PM Joey Frazee wrote: > > +1 (binding) > > - Verified checksums and signatures > - Did full build in Java 11 on Fedora > - Ran build with rpm profile and tested rpm install >

  1   2   3   4   5   6   7   >