Matt/David, By this evening, I should be at a point where I can share my branch. It should be far enough along that y'all can see what I mean about how most of the changes really weren't that complicated. My sense is that if we collaborate on it, we can probably get it ready for a PR within a week or two.
It would probably be a good idea to plan to revisit the Cassandra DMC's design and make it more flexible. One nice thing about the new DataStax driver is that it supports configuration by a very detailed configuration file format, so we can give users that option + combine it with EL/parameters (I envision an option where the user puts EL in the file, we load the file, preprocess the EL and load that into the driver) On Wed, Mar 20, 2024 at 4:01 PM Mike Thomsen <mikerthom...@gmail.com> wrote: > If it were that simple, they would probably have just gone with that > solution. That said, the API is functionally vendor agnostic at this point > at the Java API level. So I see no need to add abstraction above that. I've > got probably 2/3 of nifi-cassandra-bundle converted. Hitting a few pain > points where I'm having to dig deep into the docs to make progress, but so > far, so good. > > On Wed, Mar 20, 2024 at 2:38 PM Matt Burgess <mattyb...@apache.org> wrote: > >> It would be interesting to see if you exclude the Scylla API JAR from the >> Scylla implementation and instead include DataStax's, if that works. >> However I'm still leaning towards a vendor-agnostic API. >> >> On Wed, Mar 20, 2024 at 11:26 AM Mike Thomsen <mikerthom...@gmail.com> >> wrote: >> >> > At first glance, the package names look identical to me: >> > >> > https://java-driver.docs.scylladb.com/scylla-4.15.0.x/api/index.html >> > >> > So I see no reason to not take them at their word that it's drop-in >> > >> > On Wed, Mar 20, 2024 at 11:04 AM David Handermann < >> > exceptionfact...@apache.org> wrote: >> > >> > > Mike, >> > > >> > > One important thing to mention about the DataStax vs ScyllaDB driver >> > > is that the Maven coordinates are different, and managing the >> > > dependencies correctly will make or break the implementation. >> > > >> > > In other words, if it is possible to use the DataStax 4 core JAR in >> > > the Controller Service API, but use the ScyllaDB 3 query JAR in the >> > > ScyllaDB implementation, then that could avoid the need for additional >> > > abstraction. Without taking a closer look, however, I would be >> > > surprised if this worked. >> > > >> > > Although ScyllaDB highlights their forked driver as a drop-in >> > > replacement for the DataStax version, and maintains the same Java >> > > package names, there is a difference between a complete replacement >> > > and a shared API JAR. Without a common API JAR, that both >> > > implementations can use, it will be necessary to provide an >> > > abstraction in NiFi that avoids depending on either library at the >> > > Controller Service API level. >> > > >> > > Regards, >> > > David Handermann >> > > >> > > On Wed, Mar 20, 2024 at 8:25 AM Mike Thomsen <mikerthom...@gmail.com> >> > > wrote: >> > > > >> > > > Matt/David, >> > > > >> > > > I got some time this morning to take a crack at directly migrating >> it >> > > over >> > > > to the DataStax 4.17 driver. Definitely got a lot of work to do, >> but so >> > > far >> > > > I haven't hit any real snags. This is a branch that reverts the >> commit >> > to >> > > > remove the cassandra bundle and reuses the existing features as a >> > > > foundation. From what I'm seeing so far (feels like I'm about 25% >> done) >> > > it >> > > > should be doable to reuse the existing bundle, but rename it to the >> > "CQL >> > > > Bundle" and just add a second controller service for Scylla that is >> > > > otherwise 100% the same codewise. >> > > > >> > > > On Tue, Mar 19, 2024 at 6:41 PM Mike Thomsen < >> mikerthom...@gmail.com> >> > > wrote: >> > > > >> > > > > A cursory look at the Cassandra 5 stuff didn’t indicate any >> > > > > incompatibility. So yeah, I think we are likely pretty safe to use >> > the >> > > 4.17 >> > > > > driver >> > > > > Sent from my iPhone >> > > > > >> > > > > > On Mar 19, 2024, at 3:35 PM, Matt Burgess <mattyb...@apache.org >> > >> > > wrote: >> > > > > > >> > > > > > Is it likely now (due to the refactor) that we will simply be >> able >> > > to >> > > > > > upgrade the driver when Cassandra 5 is GA? Also does anyone use >> > > Netflix's >> > > > > > Astyanax [1]? >> > > > > > >> > > > > > [1] >> > > > > > >> > > > > >> > > >> > >> https://cassandra.apache.org/doc/stable/cassandra/getting_started/drivers.html#java >> > > > > > >> > > > > >> On Tue, Mar 19, 2024 at 3:10 PM Mike Thomsen < >> > > mikerthom...@gmail.com> >> > > > > wrote: >> > > > > >> >> > > > > >> Realistically, I think we are only likely to see two drivers: >> > > > > >> >> > > > > >> * DataStax >> > > > > >> * ScyllaDB >> > > > > >> >> > > > > >> The latter makes a selling point of being a binary compatible, >> > > drop-in >> > > > > >> replacement for the former. >> > > > > >> >> > > > > >> That's why I don't see a need to have an abstraction layer per >> > se. I >> > > > > think >> > > > > >> we only need "DataStaxConnectionProviderImpl" and >> > > > > >> "ScyllaDBConnectionProviderImpl" with the difference being >> which >> > > jar is >> > > > > >> imported by maven. >> > > > > >> >> > > > > >> On Tue, Mar 19, 2024 at 2:59 PM David Handermann < >> > > > > >> exceptionfact...@apache.org> wrote: >> > > > > >> >> > > > > >>> Mike, >> > > > > >>> >> > > > > >>> Thanks for the reply and clarification. >> > > > > >>> >> > > > > >>> I agree there is no need to maintain support for the DataStax >> 3 >> > > driver >> > > > > >>> and Java API, any new components should be built on the latest >> > > version >> > > > > >>> of the driver. >> > > > > >>> >> > > > > >>> What we do need going forward is to avoid, if at all possible, >> > > having >> > > > > >>> a DataStax 4 dependency in the Controller Service API. >> > > > > >>> >> > > > > >>> One example of this is the WebClientServiceProvider interface. >> > That >> > > > > >>> Controller Service API does not have any third-party >> > dependencies. >> > > The >> > > > > >>> Controller Service implementation, >> > > StandardWebClientServiceProvider, >> > > > > >>> has a dependency on OkHttp to implement HTTP communication. >> That >> > is >> > > > > >>> the kind of abstraction that would be ideal, and I believe >> that >> > > also >> > > > > >>> aligns with what Matt has described. >> > > > > >>> >> > > > > >>> Regards, >> > > > > >>> David Handermann >> > > > > >>> >> > > > > >>> On Tue, Mar 19, 2024 at 1:45 PM Mike Thomsen < >> > > mikerthom...@gmail.com> >> > > > > >>> wrote: >> > > > > >>>> >> > > > > >>>> ** we can dump v3 **DRIVER** compatibility, since later 4.X >> Java >> > > > > >> drivers >> > > > > >>>> are backward compatible with Cassandra 3 >> > > > > >>>> >> > > > > >>>> On Tue, Mar 19, 2024 at 2:43 PM Mike Thomsen < >> > > mikerthom...@gmail.com> >> > > > > >>> wrote: >> > > > > >>>> >> > > > > >>>>> David, >> > > > > >>>>> >> > > > > >>>>> Before we proceed, I think we should make sure we're all >> > > > > >> understanding >> > > > > >>> the >> > > > > >>>>> same problem here. Starting with this: >> > > > > >>>>> >> > > > > >>>>>> I believe the CQL protocol is backwards compatible but the >> > Java >> > > API >> > > > > >>> is >> > > > > >>>>> not. >> > > > > >>>>>> For example "com.datastax.driver.core.Session" is now >> > > > > >>>>>> "com.datastax.oss.driver.api.core.session.Session" and >> there >> > is >> > > no >> > > > > >>> more >> > > > > >>>>>> "Cluster" class. Might be fairly trivial to fix though, if >> > > that's >> > > > > >> the >> > > > > >>>>> path >> > > > > >>>>>> of least resistance. >> > > > > >>>>> >> > > > > >>>>> From what I've learned using Cassandra 3 and 4 in my day job >> > and >> > > > > >>> reading >> > > > > >>>>> up on this stuff for the sake of discussion, that all >> tracks. >> > We >> > > used >> > > > > >>> the >> > > > > >>>>> ~4.11 driver in Spring Boot on both v3 and v4 clusters >> without >> > > issue >> > > > > >>> during >> > > > > >>>>> an upgrade. So I don't see any reason to factor in the >> "changes >> > > from >> > > > > >>>>> DataStax 3 to 4" since the changes were likely a one-off >> > decision >> > > > > >>> meant to >> > > > > >>>>> position the driver for better future support and stability. >> > > > > >>>>> >> > > > > >>>>> TL;DR, we can dump v3 compatibility and the only thing our >> > users >> > > will >> > > > > >>>>> notice is if we make the controller service totally >> > incompatible >> > > with >> > > > > >>> the >> > > > > >>>>> one they're already using which is something we can actively >> > > avoid. >> > > > > >>>>> >> > > > > >>>>> On Tue, Mar 19, 2024 at 2:00 PM David Handermann < >> > > > > >>>>> exceptionfact...@apache.org> wrote: >> > > > > >>>>> >> > > > > >>>>>> All, >> > > > > >>>>>> >> > > > > >>>>>> I support a Controller Service API abstraction around the >> > > Cassandra >> > > > > >>>>>> Driver. The changes from DataStax 3 to 4 already highlight >> the >> > > need >> > > > > >>>>>> for that abstraction. The donation of the DataStax Java >> driver >> > > to >> > > > > >>>>>> Apache [1] also shows the value of providing some level of >> > > > > >> isolation, >> > > > > >>>>>> if at all possible. >> > > > > >>>>>> >> > > > > >>>>>> I have not taken a close look at the Matt's branch, and the >> > > details >> > > > > >> of >> > > > > >>>>>> the abstraction are important, but having the abstraction >> can >> > be >> > > > > >>>>>> useful to avoid getting back to this same situation. >> > > > > >>>>>> >> > > > > >>>>>> Regards, >> > > > > >>>>>> David Handermann >> > > > > >>>>>> >> > > > > >>>>>> [1] https://github.com/apache/cassandra-java-driver/ >> > > > > >>>>>> >> > > > > >>>>>> On Tue, Mar 19, 2024 at 12:37 PM Mike Thomsen < >> > > > > >> mikerthom...@gmail.com >> > > > > >>>> >> > > > > >>>>>> wrote: >> > > > > >>>>>>> >> > > > > >>>>>>> Matt, >> > > > > >>>>>>> >> > > > > >>>>>>> I got that. My point was that the Java changes appear to >> be a >> > > one >> > > > > >>> time >> > > > > >>>>>>> thing that DataStax did to make a better driver with a >> much >> > > more >> > > > > >>>>>>> future-proof API. Since Scylla tracks them as closely as >> > > > > >> possible, I >> > > > > >>>>>>> suspect that we don't need to plan for a bunch of >> abstraction >> > > to >> > > > > >>> isolate >> > > > > >>>>>>> Java changes. >> > > > > >>>>>>> >> > > > > >>>>>>> On Tue, Mar 19, 2024 at 11:07 AM Steven Matison < >> > > > > >>>>>> steven.mati...@gmail.com> >> > > > > >>>>>>> wrote: >> > > > > >>>>>>> >> > > > > >>>>>>>> That was kinda where i got stuck and fell out on my >> > > branch/jira. >> > > > > >>>>>> Mike and >> > > > > >>>>>>>> I wanted to make a new controller service , without >> backward >> > > > > >>>>>> compatibility; >> > > > > >>>>>>>> and remove the duplicate driver/connection properties >> found >> > in >> > > > > >>> some >> > > > > >>>>>> of the >> > > > > >>>>>>>> processors. >> > > > > >>>>>>>> >> > > > > >>>>>>>> I agree taking out all old stuff and making new >> controller >> > > > > >> service >> > > > > >>>>>> makes >> > > > > >>>>>>>> most sense. 4.x and 5.x should be mostly backwards >> > compatible >> > > > > >> to >> > > > > >>>>>> 2&3.x >> > > > > >>>>>>>> with how it’s used within current processors. >> > > > > >>>>>>>> >> > > > > >>>>>>>> >> > > > > >>>>>>>> >> > > > > >>>>>>>> On Tue, Mar 19, 2024 at 10:49 AM Matt Burgess < >> > > > > >>> mattyb...@apache.org> >> > > > > >>>>>>>> wrote: >> > > > > >>>>>>>> >> > > > > >>>>>>>>> The abstraction is to isolate Java API changes, not >> > protocol >> > > > > >>>>>>>> compatibility >> > > > > >>>>>>>>> Changing to the java-driver comes with a number of >> changes >> > to >> > > > > >>> the >> > > > > >>>>>> code >> > > > > >>>>>>>> (see >> > > > > >>>>>>>>> Steven's and my branches), if we can abstract that API >> it >> > > > > >> should >> > > > > >>>>>> lead to >> > > > > >>>>>>>>> more maintainable code in the future by not having to >> > change >> > > > > >> any >> > > > > >>>>>>>>> processors, just the controller service implementation. >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> >> > > > > >>>>>>>>> On Tue, Mar 19, 2024 at 10:14 AM Mike Thomsen < >> > > > > >>>>>> mikerthom...@gmail.com> >> > > > > >>>>>>>>> wrote: >> > > > > >>>>>>>>> >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>> >> > > > > >>>>>>>> >> > > > > >>>>>> >> > > > > >>> >> > > > > >> >> > > > > >> > > >> > >> https://opensource.docs.scylladb.com/stable/using-scylla/drivers/cql-drivers/scylla-java-driver.html >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> Directly quoting Scylla docs here: >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>>> The Scylla Java Driver is a drop-in replacement for >> the >> > > > > >>>>>> DataStax Java >> > > > > >>>>>>>>>> Driver. As such, no code changes are needed to use this >> > > > > >>> driver. >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>> On Tue, Mar 19, 2024 at 10:13 AM Mike Thomsen < >> > > > > >>>>>> mikerthom...@gmail.com> >> > > > > >>>>>>>>>> wrote: >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>>>> Matt, >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>>> I don't think we need to really "abstract above" the >> > > > > >> drivers >> > > > > >>>>>> because >> > > > > >>>>>>>>> the >> > > > > >>>>>>>>>>> Java DataStax driver appears to support 4.X all the >> way >> > > > > >>> back to >> > > > > >>>>>> 2.X, >> > > > > >>>>>>>> as >> > > > > >>>>>>>>>>> well as the enterprise versions from DataStax >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>>> >> > > > > >>>>>> >> > > https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>>> Similar situation with Scylla. When I looked at the >> > > > > >> driver, >> > > > > >>> it >> > > > > >>>>>>>> appeared >> > > > > >>>>>>>>>> to >> > > > > >>>>>>>>>>> copy verbatim the entire public API of that driver. >> So I >> > > > > >>> think >> > > > > >>>>>> before >> > > > > >>>>>>>>> we >> > > > > >>>>>>>>>>> dive into abstractions, it's worth doing a bit more >> > > > > >>> validation >> > > > > >>>>>> of >> > > > > >>>>>>>> these >> > > > > >>>>>>>>>>> details. IMHO, this might be a much lighter lift than >> > > > > >>>>>> anticipated. >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>>> On Mon, Mar 18, 2024 at 4:30 PM Matt Burgess < >> > > > > >>>>>> mattyb...@gmail.com> >> > > > > >>>>>>>>>> wrote: >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>>>> Totally agree, that's what my branch does (see link >> in >> > > > > >>> previous >> > > > > >>>>>>>>> email). >> > > > > >>>>>>>>>>>> The >> > > > > >>>>>>>>>>>> more I work with it, the more I think I can abstract >> it >> > > > > >>>>>> further from >> > > > > >>>>>>>>>> their >> > > > > >>>>>>>>>>>> JDBC-like API but I started with a bunch of delegate >> > > > > >>> classes >> > > > > >>>>>> then I >> > > > > >>>>>>>>>> figure >> > > > > >>>>>>>>>>>> I'll see where I can consolidate to more abstract >> > > > > >>> concepts. If >> > > > > >>>>>> I >> > > > > >>>>>>>> don't >> > > > > >>>>>>>>>>>> have >> > > > > >>>>>>>>>>>> to support Cassandra 3 with the new API, so much the >> > > > > >>> better. >> > > > > >>>>>>>>>>>> >> > > > > >>>>>>>>>>>> Regards, >> > > > > >>>>>>>>>>>> Matt >> > > > > >>>>>>>>>>>> >> > > > > >>>>>>>>>>>> On Mon, Mar 18, 2024 at 4:14 PM David Handermann < >> > > > > >>>>>>>>>>>> exceptionfact...@apache.org> wrote: >> > > > > >>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> Matt et al, >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> It is good to see the background effort on moving >> > > > > >>> Cassandra >> > > > > >>>>>>>>>>>>> capabilities in a supportable direction. >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> I think new Cassandra components will require a >> > > > > >>> significant >> > > > > >>>>>>>>> departure >> > > > > >>>>>>>>>>>>> from current Controller Service abstractions. Right >> > > > > >> now, >> > > > > >>> the >> > > > > >>>>>>>>> existing >> > > > > >>>>>>>>>>>>> service interface does not provide a clean >> abstraction >> > > > > >>> from >> > > > > >>>>>> the >> > > > > >>>>>>>>>>>>> Cassandra library, which is part of the reason for >> the >> > > > > >>>>>> current >> > > > > >>>>>>>>>>>>> coupling to the legacy driver version. >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> Following up from Joe's comments, it seems like the >> > > > > >>> cleanest >> > > > > >>>>>> way >> > > > > >>>>>>>>>>>>> forward is to deprecate the current bundle on the >> 1.x >> > > > > >>>>>> branch, and >> > > > > >>>>>>>>>>>>> remove the current bundle from the main branch. That >> > > > > >> will >> > > > > >>>>>> provide >> > > > > >>>>>>>> a >> > > > > >>>>>>>>>>>>> clean slate for new Service and Processor >> > > > > >>> implementations, >> > > > > >>>>>> without >> > > > > >>>>>>>>>>>>> concern for uncertain compatibility questions. >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> Regards, >> > > > > >>>>>>>>>>>>> David Handermann >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess < >> > > > > >>>>>>>> mattyb...@apache.org> >> > > > > >>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>> What do y'all think about removing the individual >> > > > > >>>>>> connection >> > > > > >>>>>>>>>>>> properties >> > > > > >>>>>>>>>>>>>> from the Cassandra processors for NiFi 2.0 and >> > > > > >>> requiring a >> > > > > >>>>>>>>>>>>>> CassandraSessionProvider instead? I think we >> started >> > > > > >>> doing >> > > > > >>>>>> that >> > > > > >>>>>>>>>>>> elsewhere >> > > > > >>>>>>>>>>>>>> (Elasticsearch maybe?), I noticed duplicate code in >> > > > > >> the >> > > > > >>>>>>>>>>>>>> CassandraSessionProvider and >> > > > > >>> AbstractCassandraProcessor, >> > > > > >>>>>> if we >> > > > > >>>>>>>>> keep >> > > > > >>>>>>>>>>>> those >> > > > > >>>>>>>>>>>>>> properties I can refactor them into a utility >> class. >> > > > > >>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>> Thanks, >> > > > > >>>>>>>>>>>>>> Matt >> > > > > >>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 2:44 PM Steven Matison < >> > > > > >>>>>>>>>>>> steven.mati...@gmail.com >> > > > > >>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> I got through quite a bit of work to enable 4.x… >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> The 3.x pieces that were not backwards compatible >> > > > > >> is >> > > > > >>>>>> very edge >> > > > > >>>>>>>>> use >> > > > > >>>>>>>>>>>>> case and >> > > > > >>>>>>>>>>>>>>> could have been done slightly differently but with >> > > > > >>> work >> > > > > >>>>>>>> around. >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>> https://github.com/steven-matison/nifi/tree/nifi-10120-1 >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess < >> > > > > >>>>>>>>>> mattyb...@apache.org> >> > > > > >>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>> Oops used the wrong email address so if there >> > > > > >> have >> > > > > >>> been >> > > > > >>>>>>>>>> responses >> > > > > >>>>>>>>>>>> to >> > > > > >>>>>>>>>>>>> the >> > > > > >>>>>>>>>>>>>>>> Cassandra thread since mine I missed them, my >> > > > > >> bad! >> > > > > >>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess < >> > > > > >>>>>>>>>> mattyb...@gmail.com >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>> I believe the CQL protocol is backwards >> > > > > >>> compatible >> > > > > >>>>>> but the >> > > > > >>>>>>>>>> Java >> > > > > >>>>>>>>>>>>> API is >> > > > > >>>>>>>>>>>>>>>>> not. For example >> > > > > >>> "com.datastax.driver.core.Session" >> > > > > >>>>>> is now >> > > > > >>>>>>>>>>>>>>>>> >> > > > > >>> "com.datastax.oss.driver.api.core.session.Session" >> > > > > >>>>>> and >> > > > > >>>>>>>> there >> > > > > >>>>>>>>>> is >> > > > > >>>>>>>>>>>> no >> > > > > >>>>>>>>>>>>> more >> > > > > >>>>>>>>>>>>>>>>> "Cluster" class. Might be fairly trivial to fix >> > > > > >>>>>> though, if >> > > > > >>>>>>>>>>>> that's >> > > > > >>>>>>>>>>>>> the >> > > > > >>>>>>>>>>>>>>>> path >> > > > > >>>>>>>>>>>>>>>>> of least resistance. >> > > > > >>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 1:40 PM Joe Witt < >> > > > > >>>>>>>>> joe.w...@gmail.com> >> > > > > >>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>> Matt >> > > > > >>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>> I dont know a ton about Cassandra but when I >> > > > > >>> looked >> > > > > >>>>>> at >> > > > > >>>>>>>>>>>>> client/driver >> > > > > >>>>>>>>>>>>>>>> notes >> > > > > >>>>>>>>>>>>>>>>>> for 4+ it said it was compatible all the way >> > > > > >>> back >> > > > > >>>>>> to 3.x. >> > > > > >>>>>>>>>> Not >> > > > > >>>>>>>>>>>>> sure >> > > > > >>>>>>>>>>>>>>>> what >> > > > > >>>>>>>>>>>>>>>>>> that means but it surely seems worth >> > > > > >> exploring. >> > > > > >>>>>> Also I >> > > > > >>>>>>>>> dont >> > > > > >>>>>>>>>>>> know >> > > > > >>>>>>>>>>>>> if >> > > > > >>>>>>>>>>>>>>> the >> > > > > >>>>>>>>>>>>>>>>>> 4.x drivers get rid of the vulnerable bits. >> > > > > >>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>> Thanks >> > > > > >>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess >> > > > > >> < >> > > > > >>>>>>>>>>>>> mattyb...@apache.org> >> > > > > >>>>>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>> At the very least we should upgrade to >> > > > > >>> Cassandra >> > > > > >>>>>>>> 3.11.6: >> > > > > >>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > > >>>>>> >> > > > > >>> >> > > >> https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt >> > > > > >>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 1:31 PM Matt >> > > > > >> Burgess < >> > > > > >>>>>>>>>>>>> mattyb...@apache.org> >> > > > > >>>>>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>> If the community agrees to get rid of >> > > > > >>> Cassandra >> > > > > >>>>>> 3 >> > > > > >>>>>>>>> that'll >> > > > > >>>>>>>>>>>>> save me >> > > > > >>>>>>>>>>>>>>>>>> effort >> > > > > >>>>>>>>>>>>>>>>>>>> on the refactor after I add Cassandra 4 :) >> > > > > >>>>>> Otherwise >> > > > > >>>>>>>>>> those >> > > > > >>>>>>>>>>>>>>>>>>>> vulnerabilities would only be in a "new" >> > > > > >>>>>> Cassandra 3 >> > > > > >>>>>>>>>>>> services >> > > > > >>>>>>>>>>>>> NAR >> > > > > >>>>>>>>>>>>>>>> that >> > > > > >>>>>>>>>>>>>>>>>>>> would not be included in the convenience >> > > > > >>> binary. >> > > > > >>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 1:28 PM Joe Witt < >> > > > > >>>>>>>>>>>> joe.w...@gmail.com> >> > > > > >>>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> Mike, Matt, >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> Happy to hear you both have active >> > > > > >> efforts >> > > > > >>> or >> > > > > >>>>>> are >> > > > > >>>>>>>>>>>> interested >> > > > > >>>>>>>>>>>>> in >> > > > > >>>>>>>>>>>>>>>> doing >> > > > > >>>>>>>>>>>>>>>>>>> so. >> > > > > >>>>>>>>>>>>>>>>>>>>> Can you help me understand more >> > > > > >>> specifically >> > > > > >>>>>> what >> > > > > >>>>>>>> that >> > > > > >>>>>>>>>>>> means >> > > > > >>>>>>>>>>>>> for >> > > > > >>>>>>>>>>>>>>>> the >> > > > > >>>>>>>>>>>>>>>>>>>>> current set of components? >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> The CVE hits are concerning and long >> > > > > >>> standing. >> > > > > >>>>>>>>>> Supporting >> > > > > >>>>>>>>>>>>>>>> Cassandra >> > > > > >>>>>>>>>>>>>>>>>> 3 >> > > > > >>>>>>>>>>>>>>>>>>>>> implies the current set of dependencies >> > > > > >>> would >> > > > > >>>>>> remain >> > > > > >>>>>>>>> too >> > > > > >>>>>>>>>>>>> right? >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> Is the current set of components we have >> > > > > >>> ones >> > > > > >>>>>> we >> > > > > >>>>>>>> want >> > > > > >>>>>>>>> to >> > > > > >>>>>>>>>>>>> retain? >> > > > > >>>>>>>>>>>>>>>> We >> > > > > >>>>>>>>>>>>>>>>>>>>> certainly need Cassandra components - but >> > > > > >>> are >> > > > > >>>>>> the >> > > > > >>>>>>>> ones >> > > > > >>>>>>>>>> we >> > > > > >>>>>>>>>>>>> have >> > > > > >>>>>>>>>>>>>>> now >> > > > > >>>>>>>>>>>>>>>>>> the >> > > > > >>>>>>>>>>>>>>>>>>>>> right ones? >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> Thanks >> > > > > >>>>>>>>>>>>>>>>>>>>> Joe >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 10:25 AM Matt >> > > > > >>> Burgess < >> > > > > >>>>>>>>>>>>>>>> mattyb...@apache.org> >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>> I'm actively working this, I pushed my >> > > > > >>>>>> branch up >> > > > > >>>>>>>> in >> > > > > >>>>>>>>>> case >> > > > > >>>>>>>>>>>>> anyone >> > > > > >>>>>>>>>>>>>>>>>> wants >> > > > > >>>>>>>>>>>>>>>>>>> to >> > > > > >>>>>>>>>>>>>>>>>>>>>> take a look [1]. The idea is to >> > > > > >> abstract >> > > > > >>> the >> > > > > >>>>>>>>> Cassandra >> > > > > >>>>>>>>>>>> API >> > > > > >>>>>>>>>>>>> "up >> > > > > >>>>>>>>>>>>>>> a >> > > > > >>>>>>>>>>>>>>>>>>> couple >> > > > > >>>>>>>>>>>>>>>>>>>>>> levels" and provide implementations for >> > > > > >>>>>> Cassandra >> > > > > >>>>>>>> 3, >> > > > > >>>>>>>>>> 4, >> > > > > >>>>>>>>>>>> and >> > > > > >>>>>>>>>>>>>>>>>> eventually >> > > > > >>>>>>>>>>>>>>>>>>>>> 5. >> > > > > >>>>>>>>>>>>>>>>>>>>>> For JDBC-like interfaces this is a PITA >> > > > > >>>>>> because of >> > > > > >>>>>>>>> the >> > > > > >>>>>>>>>>>> API >> > > > > >>>>>>>>>>>>>>>>>> (Statement, >> > > > > >>>>>>>>>>>>>>>>>>>>>> PreparedStatement, BoundStatement, >> > > > > >>> ResultSet, >> > > > > >>>>>>>> etc.) >> > > > > >>>>>>>>>> but >> > > > > >>>>>>>>>>>> I'm >> > > > > >>>>>>>>>>>>>>>> hoping >> > > > > >>>>>>>>>>>>>>>>>> we >> > > > > >>>>>>>>>>>>>>>>>>>>> can >> > > > > >>>>>>>>>>>>>>>>>>>>>> find a common pattern for abstracting >> > > > > >> the >> > > > > >>>>>>>>> third-party >> > > > > >>>>>>>>>>>>> library >> > > > > >>>>>>>>>>>>>>>>>>>>>> implementation and API from the NiFi >> > > > > >>>>>> component >> > > > > >>>>>>>>>>>> (Processor, >> > > > > >>>>>>>>>>>>>>>>>>>>>> ControllerService, etc.) API. I think >> > > > > >>> we're >> > > > > >>>>>> doing >> > > > > >>>>>>>>>>>> something >> > > > > >>>>>>>>>>>>>>>> similar >> > > > > >>>>>>>>>>>>>>>>>>> for >> > > > > >>>>>>>>>>>>>>>>>>>>>> Kafka? >> > > > > >>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>> Regards, >> > > > > >>>>>>>>>>>>>>>>>>>>>> Matt >> > > > > >>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>> [1] >> > > > > >>>>>> https://github.com/mattyb149/nifi/tree/cassy4 >> > > > > >>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>> On Fri, Mar 15, 2024 at 8:43 AM Mike >> > > > > >>> Thomsen >> > > > > >>>>>> < >> > > > > >>>>>>>>>>>>>>>>>> mikerthom...@gmail.com> >> > > > > >>>>>>>>>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>> That’s been on my todo list for a >> > > > > >>> little >> > > > > >>>>>> while >> > > > > >>>>>>>> but >> > > > > >>>>>>>>>>>> things >> > > > > >>>>>>>>>>>>>>> kept >> > > > > >>>>>>>>>>>>>>>>>>> coming >> > > > > >>>>>>>>>>>>>>>>>>>>> up. >> > > > > >>>>>>>>>>>>>>>>>>>>>>> I think I could get started on that >> > > > > >>> now. >> > > > > >>>>>> Based >> > > > > >>>>>>>> on >> > > > > >>>>>>>>> my >> > > > > >>>>>>>>>>>>> initial >> > > > > >>>>>>>>>>>>>>>>>>> research >> > > > > >>>>>>>>>>>>>>>>>>>>> it >> > > > > >>>>>>>>>>>>>>>>>>>>>>> appears that scylla uses the exact >> > > > > >> same >> > > > > >>>>>> api as >> > > > > >>>>>>>>>>>> datastax >> > > > > >>>>>>>>>>>>> so >> > > > > >>>>>>>>>>>>>>>>>>> supporting >> > > > > >>>>>>>>>>>>>>>>>>>>>> both >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in a cql bundle should theoretically >> > > > > >> be >> > > > > >>>>>> fairly >> > > > > >>>>>>>>> easy. >> > > > > >>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Sent from my iPhone >> > > > > >>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Mar 14, 2024, at 6:18 PM, Joe >> > > > > >>> Witt < >> > > > > >>>>>>>>>>>>> joew...@apache.org> >> > > > > >>>>>>>>>>>>>>>>>> wrote: >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Team, >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Cassandra remains a really >> > > > > >> important >> > > > > >>>>>> system to >> > > > > >>>>>>>>> be >> > > > > >>>>>>>>>>>> able >> > > > > >>>>>>>>>>>>> to >> > > > > >>>>>>>>>>>>>>>> send >> > > > > >>>>>>>>>>>>>>>>>>> data >> > > > > >>>>>>>>>>>>>>>>>>>>> to. >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, it seems like we've not >> > > > > >>>>>> maintained >> > > > > >>>>>>>>> these >> > > > > >>>>>>>>>>>>> well. We >> > > > > >>>>>>>>>>>>>>>>>> have >> > > > > >>>>>>>>>>>>>>>>>>>>> what >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> appears to be at least a full >> > > > > >>> generation >> > > > > >>>>>>>> behind >> > > > > >>>>>>>>> on >> > > > > >>>>>>>>>>>>> client >> > > > > >>>>>>>>>>>>>>>>>> versions >> > > > > >>>>>>>>>>>>>>>>>>>>> (we >> > > > > >>>>>>>>>>>>>>>>>>>>>>> are >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> on 3x vs 4x which is the latest >> > > > > >>> stable >> > > > > >>>>>> with 5x >> > > > > >>>>>>>>>>>>> apparently >> > > > > >>>>>>>>>>>>>>>>>> coming >> > > > > >>>>>>>>>>>>>>>>>>>>>>> shortly). >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> We have components to send data, >> > > > > >>> query >> > > > > >>>>>> data, >> > > > > >>>>>>>> and >> > > > > >>>>>>>>>> use >> > > > > >>>>>>>>>>>>>>>> Cassandra >> > > > > >>>>>>>>>>>>>>>>>> as >> > > > > >>>>>>>>>>>>>>>>>>> a >> > > > > >>>>>>>>>>>>>>>>>>>>>> cache >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> store. We have older mechanisms >> > > > > >> for >> > > > > >>>>>> json/avro >> > > > > >>>>>>>>> and >> > > > > >>>>>>>>>>>>> publish >> > > > > >>>>>>>>>>>>>>>>>>>>> mechanisms >> > > > > >>>>>>>>>>>>>>>>>>>>>> for >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> records. >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> The libraries we do have depend on >> > > > > >>>>>> outdated >> > > > > >>>>>>>>>>>> versions of >> > > > > >>>>>>>>>>>>>>> Guava >> > > > > >>>>>>>>>>>>>>>>>> and >> > > > > >>>>>>>>>>>>>>>>>>>>>> result >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> many CVE hits. >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> I am inclined to think we should >> > > > > >>>>>> deprecate the >> > > > > >>>>>>>>> 1.x >> > > > > >>>>>>>>>>>>>>> components >> > > > > >>>>>>>>>>>>>>>>>> and >> > > > > >>>>>>>>>>>>>>>>>>>>>> remove >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> them as-is from the 2.x line. Then >> > > > > >>>>>>>> re-introduce >> > > > > >>>>>>>>>>>> them >> > > > > >>>>>>>>>>>>> with >> > > > > >>>>>>>>>>>>>>>>>> record >> > > > > >>>>>>>>>>>>>>>>>>>>> only >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> interfaces and built against the >> > > > > >>> latest >> > > > > >>>>>> stable >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Cassandra/Datastax/ScyllaDB >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> interfaces. >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> I'd love to hear thoughts from >> > > > > >> those >> > > > > >>>>>> closer to >> > > > > >>>>>>>>>> this >> > > > > >>>>>>>>>>>>> space >> > > > > >>>>>>>>>>>>>>>> both >> > > > > >>>>>>>>>>>>>>>>>> as >> > > > > >>>>>>>>>>>>>>>>>>> a >> > > > > >>>>>>>>>>>>>>>>>>>>>> user >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> and developer so we can make good >> > > > > >>> next >> > > > > >>>>>> steps. >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks >> > > > > >>>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>>>> >> > > > > >>>>>>>>>>>>> >> > > > > >>>>>>>>>>>> >> > > > > >>>>>>>>>>> >> > > > > >>>>>>>>>> >> > > > > >>>>>>>>> >> > > > > >>>>>>>> >> > > > > >>>>>> >> > > > > >>>>> >> > > > > >>> >> > > > > >> >> > > > > >> > > >> > >> >
