Great stuff. Keep it going. If we go to replace this then posts like this are great to gather the feedback.
I like the protobuf idea. If we were on Maven then we could use the protobuf plugin which compiles the schema as part of the build and it generates Java classes from it which we might use for actual query serialization into binary format. Unfortunately there is nothing like that for Ant, protoc would need to be a local dependency on the computer which compiles the project to be able to do that so that is kind of a dead end. Or is there any workaround here? On Thu, Sep 19, 2024 at 8:14 PM David Capwell <dcapw...@apple.com> wrote: > I personally don’t mind switching off Chronicle Queue. I have a > transformer function to convert the FQL logs to Thrift (don’t judge) and > use easy-cas to reply on a cluster… replying FQL from Chronicle Queue was > far slower than Thrift and was hard to push the cluster as the client was > the bottleneck… switching off it let me actually cause Cassandra to be the > bottleneck… > > No, there is another perfectly sensible option: just implement a simple > serialisation format ourselves. > > > My one issue with this is we need to ask who the target audience is? > Trying to add FQL reply to easy-cas was a pain for 2 reasons: Chronicle > Queue is slow, custom C* serializers that must be in the class path (that > brings a ton of baggage with it)… > > For me FQL has 2 use cases > > 1) analytic, what are people actually doing and what are there frequencies? > 2) reply > > In both cases custom serializers are a pain due to baggage they bring and > the limiting nature of it… what if I want a Go based FQL reply? I need > java code from cassandra-all… > > I personally favor serializers like protobuf/thrift as they are portable > and can be used by users without issues. As for the log format itself… a > super simple log format that can be easy to read that is custom is fine by > me… I am cool with the log being custom as I don’t know a good portable log > format at the top of my head… a simple thing like the following works for me > > Header: lengths, checksum, etc. > Body: std serializer > + > > On Sep 19, 2024, at 9:14 AM, C. Scott Andreas <sc...@paradoxica.net> > wrote: > > Agree with Benedict's proposal here. > > In circumstances when I've needed to capture and work with FQL, I've found > it cumbersome to work with Chronicle. The dial-home functionality and > release process changes put it over the top for me. > > – Scott > > On Sep 19, 2024, at 8:40 AM, Josh McKenzie <jmcken...@apache.org> wrote: > > > there is another perfectly sensible option > > My apologies; I wasn't clear. *If we choose to continue to use chronicle > queue*, what I enumerated was the only logical option I saw for us. > > Altogether I think we should just move away from the library as you've > laid out here Benedict. > > On Thu, Sep 19, 2024, at 11:34 AM, Benedict wrote: > > > No, there is another perfectly sensible option: just implement a simple > serialisation format ourselves. > > I am against forking their code; that is a much higher maintenance burden > than just writing something simple ourselves. We’ve spent longer > collectively discussing and maintaining this dependency than it would take > to implement the features we use. > > I still have not heard a compelling reason we adopted it as a dependency > in the first place. > > On 19 Sep 2024, at 16:26, Josh McKenzie <jmcken...@apache.org> wrote: > > > > a jerk move, but they started it with this weird release model > > I think that's the only option given their release model and lack of > backporting bugfixes to the latest ea. Either you run tip of the spear, pay > them for bugfixes, or run what's effectively an unsupported LTS in the form > of ea. > > So doesn't seem like a jerk move to me as much as it seems like an > eventuality of their release model. > > On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote: > > I feel like a group of us discussed this IRL a bit at ApacheCon in Vegas ~ > 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone explaining > the string operations overhead in the JVM of log concatenation vs slapping > binary to CQ’s off heap-and-append operation was substantial. > > We could hostile fork and bring the bits we use in tree (a jerk move, but > they started it with this weird release model). I’d rather avoid this, but > it’s an option seeing as how it’s ASFv2. > > On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan <jeremiah.jor...@gmail.com> > wrote: > > > When it comes to alternatives, what about logback + slf4j? It has > appenders where we want, it is sync / async, we can code some nio appender > too I guess, it logs it as text into a file so we do not need any special > tooling to review that. For tailing which Chronicle also offers, I guess > "tail -f that.log" just does the job? logback even rolls the files after > they are big enough so it rolls the files the same way after some > configured period / size as Chronicle does (It even compresses the logs). > > > Yes it was considered. The whole point was to have a binary log because > serialization to/from (remember replay is part off this) text explodes the > size on disk and in memory as well as the processing time required and does > not meet the timing requirements of fqltool. > > -Jeremiah > > > > > > >
