I am not sure what you mean. I mean, I do, but not following. Look into FullQueryLogger (1) what it goes to put into CQL is a query like String wrapped in a Query object. It literally take a String as a representation of a query a user executed. We just replace this by serializing that query to protobuf. What is counter productive? We just replace one thing for another. Audit message / events would be similar.
(1) https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/fql/FullQueryLogger.java#L320-L321 On Thu, Sep 19, 2024 at 11:17 PM J. D. Jordan <jeremiah.jor...@gmail.com> wrote: > I think reserializing the payload into a new format is counter productive > to some of the performance goals of the binary logs? > If you have to deserialize and reserialize the message you are going to be > throwing off a ton of extra GC. > I think we have done a lot of work in recent version to reduce the amount > of re-serialization that happens in the query paths? Not sure we want to > add some back in on purpose? Keeping the payload in the internal > serialization format does indeed have the drawbacks David mentioned, but I > think “zero serialization overhead” is a pretty big advantage to keeping > things that way? > > -Jeremiah > > On Sep 19, 2024, at 3:56 PM, Štefan Miklošovič <smikloso...@apache.org> > wrote: > > > I think protobuf upon serialization is just a bunch of bytes anyway. If we > figure out the header as David showed then we can still serialize it all > with the machinery / serializers you mentioned. It can write bytes, right?! > I very briefly checked and I think that protobuf is super simple and does > not have any checksumming etc. so some sauce on top of that would be > necessary anyway and we can reuse what we have to produce binary files. > > On the consumer side, the binary file would be parsed with some tooling > e.g. in Go, indeed, but the headers and stuff would be so simple that it > would be just a coding exercise and then it might be deserialized with > protobuf for that language. > > Basically, only the payload itself would be the product of protobuf and > all around super simple to crack through. > > On Thu, Sep 19, 2024 at 10:41 PM Benedict <bened...@apache.org> wrote: > >> Sorry, I missed that. I’m not convinced any of these logs need language >> agnostic tools for access, but if that’s a goal for other folk I don’t feel >> strongly about it. >> >> On 19 Sep 2024, at 21:06, Štefan Miklošovič <smikloso...@apache.org> >> wrote: >> >> >> More to it, it is actually not only about FQL. Audit logging is on >> Chronicle queues too so inspecting that would be platform independent as >> well. >> >> CEP-12 suggests that there might be a persistent store for diagnostic >> events as well. If somebody wants to inspect what a node was doing after it >> went offline as for now all these events are in memory only. >> >> This would basically enable people to fully inspect what the cluster was >> doing from FQL to Audit to Diagnostics in a language independent manner. >> >> On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič <smikloso...@apache.org> >> wrote: >> >>> I think the biggest selling point for using something like protobuf is >>> what David said - what if he wants to replay it in Go? Basing it on >>> something language neutral enables people to replay it in whatever they >>> want. If we have something totally custom then it is replayable just in >>> Java without bringing tons of dependencies to their projects. That is the >>> message I got from what he wrote. >>> >>> On Thu, Sep 19, 2024 at 9:47 PM Benedict <bened...@apache.org> wrote: >>> >>>> Do we need any of these things either? We have our own serialisation >>>> framework and file readers and writers, and at least in the FQL case these >>>> are the native serialisation format. >>>> >>>> At cursory glance it also looks to me like this would be a minimal >>>> refactor from the current state. >>>> >>>> What is the reason we want to add these other dependencies? >>>> >>>> >>>> On 19 Sep 2024, at 20:31, Štefan Miklošovič <smikloso...@apache.org> >>>> wrote: >>>> >>>> >>>> well the Maven plugin declares that it downloads protoc from Maven >>>> Central automatically _somehow_ so coding up an Ant task which does >>>> something similar shouldn't be too hard. I will investigate this idea. >>>> >>>> On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams <dri...@gmail.com> >>>> wrote: >>>> >>>>> On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič >>>>> <smikloso...@apache.org> wrote: >>>>> > Unfortunately there is nothing like that for Ant, protoc would need >>>>> to be a local dependency on the computer which compiles the project to be >>>>> able to do that so that is kind of a dead end. Or is there any workaround >>>>> here? >>>>> >>>>> In the old thrift days I believe we generated the code and checked it >>>>> in so you didn't need to compile locally. >>>>> >>>>> Kind Regards, >>>>> Brandon >>>>> >>>>
