subject:"Re\: \[DISCUSS\] Chronicle Queue's development model and a hypothetical replacement of the library"

LGTM

On Mon, Sep 30, 2024, at 2:17 PM, Štefan Miklošovič wrote:
> Okay, that works, so summary:
> 
> 1) CEP-12 will be delivered WITHOUT interacting with Chronicle Queues, 
> something else will be used (most probably just a simple logger logging to a 
> text file, that is just enough for something like diagnostic logs which are 
> logged very sparsely)
> 
> 2) We will NOT use Chronicle in the future for any new implementations / 
> solutions from now on.
> 
> 3) We WILL be replacing loggers based on Chronicle Queues.
> 
> 4) A new implementation replacing Chronicle Queues SHOULD take into account a 
> heterogeneous environment when it comes to consumers of the events logged, 
> being language-agnostic or supporting an open format which enables that.
> 
> On Mon, Sep 30, 2024 at 5:07 PM Josh McKenzie  wrote:
>> __
>> I say we do CEP-12 w/out using chronicle and have a follow up JIRA to 
>> replace chronicle w/something else.
>> 
>> Seems like there's a reasonable consensus around at least that subset of 
>> things given the discussion here. As to what form that something else takes, 
>> well, that's a topic for another day. :)
>> 
>> On Mon, Sep 30, 2024, at 10:09 AM, Štefan Miklošovič wrote:
>>>  "how complex should it be to rip out the chronicle format, insert some 
>>> other well defined and well known, and handle log rolling ourselves".
>>>  
>>> I think that it will be actually easier to do after CEP-12 is in because as 
>>> I mentioned it does housekeeping of what is there rigth now and refactors 
>>> it a little bit so throwing away the guts of it should be isolated only to 
>>> actual BinLog class and stuff around that which is built on top of 
>>> Chronicle.
>>> 
>>> "There a reason we can't move forward with CEP-12 w/out addressing the 
>>> chronicle stuff? i.e."
>>> 
>>> I think there is not any.
>>> 
>>> "Why would CEP-12 be heavily coupled with chronicle?" - it would not be 
>>> heavier from what is already there for audit of fql. Chronicle here 
>>> basically acts as a sink.
>>> 
>>> Actually, that patch also makes the implementations of (diagnostic too) 
>>> loggers pluggable  (via coding against an interface and putting that on the 
>>> class path) so people might already write it to whatever they want - even 
>>> to something protobuf-like. If they do not want to use Chronicle as a sink, 
>>> by implementing their own logger, they could just put it wherever they 
>>> want. I think that I forgot to mention this aspect. So the whole solution 
>>> we have is already not hard-wired to Chronicle necessarily. It is just 
>>> something we provide out of the box.
>>> 
>>> On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie  wrote:
 __
 I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking 
 through "how complex should it be to rip out the chronicle format, insert 
 some other well defined and well known, and handle log rolling ourselves". 
 My preference (which I didn't indicate earlier) would be to have that done 
 independently.
 
 There a reason we can't move forward with CEP-12 w/out addressing the 
 chronicle stuff? i.e.
> I would like to have this resolved because there is CEP-12 I plan to 
> deliver and I hit this and I do not want to base that work on something 
> we might eventually abandon.
 Why would CEP-12 be heavily coupled with chronicle? I would assume you'd 
 be able to make light use of the existing logging + log rolling, and then 
 someone else could come along and easily rip out chronicle and the rolling 
 and add in a different one with minimal code changes?
 
 On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
> I don't understand why CEP-12 should be a vehicle for introducing changes 
> like that. That is something totally unrelated. I am not going to be the 
> one to implement anything beyond CEP-12 and I am not the one who is going 
> to replace it either so if we make it a hard requirement for CEP-12 then 
> CEP-12 as it is will never be introduced. Just want to be clear about 
> that.
> 
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  
>> wrote:
>> >
>> > I'd strongly support either rolling the format change into the CEP-12 
>> > proposal or having another CEP for introducing protobuf, spark, etc - 
>> > some kind of more broadly adopted format, and removing chronicle from 
>> > our stack.
>> 
>> +1, I too would strongly support an open format and removing chronicle.
>> 
>> Kind Regards,
>> Brandon
 
>>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

Okay, that works, so summary:

1) CEP-12 will be delivered WITHOUT interacting with Chronicle Queues,
something else will be used (most probably just a simple logger logging to
a text file, that is just enough for something like diagnostic logs which
are logged very sparsely)

2) We will NOT use Chronicle in the future for any new implementations /
solutions from now on.

3) We WILL be replacing loggers based on Chronicle Queues.

4) A new implementation replacing Chronicle Queues SHOULD take into account
a heterogeneous environment when it comes to consumers of the events
logged, being language-agnostic or supporting an open format which enables
that.

On Mon, Sep 30, 2024 at 5:07 PM Josh McKenzie  wrote:

> I say we do CEP-12 w/out using chronicle and have a follow up JIRA to
> replace chronicle w/something else.
>
> Seems like there's a reasonable consensus around at least that subset of
> things given the discussion here. As to what form that something else
> takes, well, that's a topic for another day. :)
>
> On Mon, Sep 30, 2024, at 10:09 AM, Štefan Miklošovič wrote:
>
>  "how complex should it be to rip out the chronicle format, insert some
> other well defined and well known, and handle log rolling ourselves".
>
> I think that it will be actually easier to do after CEP-12 is in because
> as I mentioned it does housekeeping of what is there rigth now and
> refactors it a little bit so throwing away the guts of it should be
> isolated only to actual BinLog class and stuff around that which is built
> on top of Chronicle.
>
> "There a reason we can't move forward with CEP-12 w/out addressing the
> chronicle stuff? i.e."
>
> I think there is not any.
>
> "Why would CEP-12 be heavily coupled with chronicle?" - it would not be
> heavier from what is already there for audit of fql. Chronicle here
> basically acts as a sink.
>
> Actually, that patch also makes the implementations of (diagnostic too)
> loggers pluggable  (via coding against an interface and putting that on the
> class path) so people might already write it to whatever they want - even
> to something protobuf-like. If they do not want to use Chronicle as a sink,
> by implementing their own logger, they could just put it wherever they
> want. I think that I forgot to mention this aspect. So the whole solution
> we have is already not hard-wired to Chronicle necessarily. It is just
> something we provide out of the box.
>
> On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie 
> wrote:
>
>
> I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking
> through "how complex should it be to rip out the chronicle format, insert
> some other well defined and well known, and handle log rolling ourselves".
> My preference (which I didn't indicate earlier) would be to have that done
> independently.
>
> There a reason we can't move forward with CEP-12 w/out addressing the
> chronicle stuff? i.e.
>
> I would like to have this resolved because there is CEP-12 I plan to
> deliver and I hit this and I do not want to base that work on something we
> might eventually abandon.
>
> Why would CEP-12 be heavily coupled with chronicle? I would assume you'd
> be able to make light use of the existing logging + log rolling, and then
> someone else could come along and easily rip out chronicle and the rolling
> and add in a different one with minimal code changes?
>
> On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
>
> I don't understand why CEP-12 should be a vehicle for introducing changes
> like that. That is something totally unrelated. I am not going to be the
> one to implement anything beyond CEP-12 and I am not the one who is going
> to replace it either so if we make it a hard requirement for CEP-12 then
> CEP-12 as it is will never be introduced. Just want to be clear about that.
>
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>
> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie 
> wrote:
> >
> > I'd strongly support either rolling the format change into the CEP-12
> proposal or having another CEP for introducing protobuf, spark, etc - some
> kind of more broadly adopted format, and removing chronicle from our stack.
>
> +1, I too would strongly support an open format and removing chronicle.
>
> Kind Regards,
> Brandon
>
>
>
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I say we do CEP-12 w/out using chronicle and have a follow up JIRA to replace 
chronicle w/something else.

Seems like there's a reasonable consensus around at least that subset of things 
given the discussion here. As to what form that something else takes, well, 
that's a topic for another day. :)

On Mon, Sep 30, 2024, at 10:09 AM, Štefan Miklošovič wrote:
>  "how complex should it be to rip out the chronicle format, insert some other 
> well defined and well known, and handle log rolling ourselves".
>  
> I think that it will be actually easier to do after CEP-12 is in because as I 
> mentioned it does housekeeping of what is there rigth now and refactors it a 
> little bit so throwing away the guts of it should be isolated only to actual 
> BinLog class and stuff around that which is built on top of Chronicle.
> 
> "There a reason we can't move forward with CEP-12 w/out addressing the 
> chronicle stuff? i.e."
> 
> I think there is not any.
> 
> "Why would CEP-12 be heavily coupled with chronicle?" - it would not be 
> heavier from what is already there for audit of fql. Chronicle here basically 
> acts as a sink. 
> 
> Actually, that patch also makes the implementations of (diagnostic too) 
> loggers pluggable  (via coding against an interface and putting that on the 
> class path) so people might already write it to whatever they want - even to 
> something protobuf-like. If they do not want to use Chronicle as a sink, by 
> implementing their own logger, they could just put it wherever they want. I 
> think that I forgot to mention this aspect. So the whole solution we have is 
> already not hard-wired to Chronicle necessarily. It is just something we 
> provide out of the box.
> 
> On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie  wrote:
>> __
>> I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking 
>> through "how complex should it be to rip out the chronicle format, insert 
>> some other well defined and well known, and handle log rolling ourselves". 
>> My preference (which I didn't indicate earlier) would be to have that done 
>> independently.
>> 
>> There a reason we can't move forward with CEP-12 w/out addressing the 
>> chronicle stuff? i.e.
>>> I would like to have this resolved because there is CEP-12 I plan to 
>>> deliver and I hit this and I do not want to base that work on something we 
>>> might eventually abandon.
>> Why would CEP-12 be heavily coupled with chronicle? I would assume you'd be 
>> able to make light use of the existing logging + log rolling, and then 
>> someone else could come along and easily rip out chronicle and the rolling 
>> and add in a different one with minimal code changes?
>> 
>> On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
>>> I don't understand why CEP-12 should be a vehicle for introducing changes 
>>> like that. That is something totally unrelated. I am not going to be the 
>>> one to implement anything beyond CEP-12 and I am not the one who is going 
>>> to replace it either so if we make it a hard requirement for CEP-12 then 
>>> CEP-12 as it is will never be introduced. Just want to be clear about that.
>>> 
>>> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
 On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  wrote:
 >
 > I'd strongly support either rolling the format change into the CEP-12 
 > proposal or having another CEP for introducing protobuf, spark, etc - 
 > some kind of more broadly adopted format, and removing chronicle from 
 > our stack.
 
 +1, I too would strongly support an open format and removing chronicle.
 
 Kind Regards,
 Brandon
>>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

 "how complex should it be to rip out the chronicle format, insert some
other well defined and well known, and handle log rolling ourselves".

I think that it will be actually easier to do after CEP-12 is in because as
I mentioned it does housekeeping of what is there rigth now and refactors
it a little bit so throwing away the guts of it should be isolated only to
actual BinLog class and stuff around that which is built on top of
Chronicle.

"There a reason we can't move forward with CEP-12 w/out addressing the
chronicle stuff? i.e."

I think there is not any.

"Why would CEP-12 be heavily coupled with chronicle?" - it would not be
heavier from what is already there for audit of fql. Chronicle here
basically acts as a sink.

Actually, that patch also makes the implementations of (diagnostic too)
loggers pluggable  (via coding against an interface and putting that on the
class path) so people might already write it to whatever they want - even
to something protobuf-like. If they do not want to use Chronicle as a sink,
by implementing their own logger, they could just put it wherever they
want. I think that I forgot to mention this aspect. So the whole solution
we have is already not hard-wired to Chronicle necessarily. It is just
something we provide out of the box.

On Mon, Sep 30, 2024 at 3:57 PM Josh McKenzie  wrote:

> I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking
> through "how complex should it be to rip out the chronicle format, insert
> some other well defined and well known, and handle log rolling ourselves".
> My preference (which I didn't indicate earlier) would be to have that done
> independently.
>
> There a reason we can't move forward with CEP-12 w/out addressing the
> chronicle stuff? i.e.
>
> I would like to have this resolved because there is CEP-12 I plan to
> deliver and I hit this and I do not want to base that work on something we
> might eventually abandon.
>
> Why would CEP-12 be heavily coupled with chronicle? I would assume you'd
> be able to make light use of the existing logging + log rolling, and then
> someone else could come along and easily rip out chronicle and the rolling
> and add in a different one with minimal code changes?
>
> On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
>
> I don't understand why CEP-12 should be a vehicle for introducing changes
> like that. That is something totally unrelated. I am not going to be the
> one to implement anything beyond CEP-12 and I am not the one who is going
> to replace it either so if we make it a hard requirement for CEP-12 then
> CEP-12 as it is will never be introduced. Just want to be clear about that.
>
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>
> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie 
> wrote:
> >
> > I'd strongly support either rolling the format change into the CEP-12
> proposal or having another CEP for introducing protobuf, spark, etc - some
> kind of more broadly adopted format, and removing chronicle from our stack.
>
> +1, I too would strongly support an open format and removing chronicle.
>
> Kind Regards,
> Brandon
>
>
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I hear you. Not trying to shoehorn a change in w/CEP-12, just thinking through 
"how complex should it be to rip out the chronicle format, insert some other 
well defined and well known, and handle log rolling ourselves". My preference 
(which I didn't indicate earlier) would be to have that done independently.

There a reason we can't move forward with CEP-12 w/out addressing the chronicle 
stuff? i.e.
> I would like to have this resolved because there is CEP-12 I plan to deliver 
> and I hit this and I do not want to base that work on something we might 
> eventually abandon.
Why would CEP-12 be heavily coupled with chronicle? I would assume you'd be 
able to make light use of the existing logging + log rolling, and then someone 
else could come along and easily rip out chronicle and the rolling and add in a 
different one with minimal code changes?

On Mon, Sep 30, 2024, at 9:15 AM, Štefan Miklošovič wrote:
> I don't understand why CEP-12 should be a vehicle for introducing changes 
> like that. That is something totally unrelated. I am not going to be the one 
> to implement anything beyond CEP-12 and I am not the one who is going to 
> replace it either so if we make it a hard requirement for CEP-12 then CEP-12 
> as it is will never be introduced. Just want to be clear about that.
> 
> On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:
>> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  wrote:
>> >
>> > I'd strongly support either rolling the format change into the CEP-12 
>> > proposal or having another CEP for introducing protobuf, spark, etc - some 
>> > kind of more broadly adopted format, and removing chronicle from our stack.
>> 
>> +1, I too would strongly support an open format and removing chronicle.
>> 
>> Kind Regards,
>> Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I don't understand why CEP-12 should be a vehicle for introducing changes
like that. That is something totally unrelated. I am not going to be the
one to implement anything beyond CEP-12 and I am not the one who is going
to replace it either so if we make it a hard requirement for CEP-12 then
CEP-12 as it is will never be introduced. Just want to be clear about that.

On Mon, Sep 30, 2024 at 3:09 PM Brandon Williams  wrote:

> On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie 
> wrote:
> >
> > I'd strongly support either rolling the format change into the CEP-12
> proposal or having another CEP for introducing protobuf, spark, etc - some
> kind of more broadly adopted format, and removing chronicle from our stack.
>
> +1, I too would strongly support an open format and removing chronicle.
>
> Kind Regards,
> Brandon
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-30 Thread Brandon Williams

On Mon, Sep 30, 2024 at 7:55 AM Josh McKenzie  wrote:
>
> I'd strongly support either rolling the format change into the CEP-12 
> proposal or having another CEP for introducing protobuf, spark, etc - some 
> kind of more broadly adopted format, and removing chronicle from our stack.

+1, I too would strongly support an open format and removing chronicle.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

> thinking more about stuff like protobuf and while I do see benefits of that, 
> honestly, it just does not matter too much if it is done like that or not.
I disagree; I think having a language and environment agnostic file format 
matters a great deal. Unless we're talking specifically about the FQL case in 
which case I totally agree. :)

Part of what makes open source projects successful is their ability to 
modularly interop with other projects, often times in ways we don't predict; us 
using the Chronicle format for our binary files is choosing a format that's 
esoteric with fairly antagonistic licensing compared to something like 
protobufs.

I'd strongly support either rolling the format change into the CEP-12 proposal 
or having another CEP for introducing protobuf, spark, etc - some kind of more 
broadly adopted format, and removing chronicle from our stack.

To Jon's point: having bespoke formats with all the code for it housed in 
cassandra-all rather than in a modular support library really ties our hands on 
a lot of fronts; updating the core DB in terms of serialization formats, of JDK 
versions, etc all potentially become client-impacting exercises.


On Mon, Sep 30, 2024, at 5:12 AM, Štefan Miklošovič wrote:
> That is all OK to mention. So as I read it correctly, Jon, myself, Andrew, 
> David - we all would like to see some cross-language events ingestion. Other 
> people do not seem to consider it important enough (just correct me if I am 
> wrong). I do not mind, I am 50:50, not an absolute must but sure, let's add 
> that to the wish list.
> 
> I believe that unless it is an absolute show-stopper (which it is not) it is 
> not necessary. It is something to keep in mind upon next refactorization. I 
> am personally not affected by this. Whoever will need that, they will find a 
> way to make it happen.
> 
> I would really appreciate it if we found consensus among these points I wrote 
> down. From my point of view, the most ideal outcome would be to make CEP-12 
> happen which cleans it all up and makes it more robust.
> 
> My perception is that we have always found the most practical solution and if 
> it brings value to a user (being able to inspect diagnostic events offline 
> for further inspection) we should not avoid delivering that. Something 
> similar happened in e.g. Password validator / generator (CEP-24) where the 
> most ideal solution was to base it on transactional configuration but even 
> though we are not there yet, that did not stop us from delivering it because 
> for some entities in this space it brought value anyway. 
> 
> It may seem as if I am invested in delivering it because I spent some time on 
> that already - that is not the case - I am OK to drop diagnostic events 
> persistence if you insist on that but honestly I just do not see any 
> compelling argument to do so. The library we are using that for (Chronicle 
> Queues) is already there, it is just a functionality addition and as I said, 
> if somebody comes and delivers a solution which would replace it, I do not 
> see it to be problematic. It would be replaced as anything else (FQL, Audit 
> ...). 
> 
> On Sun, Sep 29, 2024 at 9:06 PM Jon Haddad  wrote:
>> Strong +1 to the file format issue, and if we're building a wish list - it 
>> would be great if we could read the file format without pulling in 
>> cassandra-all.  Long term, I'd love to see this for SSTables & Commit logs 
>> as well.
>> 
>> I've long been a fan of Gradle subprojects because it makes this kind of 
>> thing fairly easy. 
>> 
>> Jon
>> 
>> 
>> 
>> On Sun, Sep 29, 2024 at 11:46 AM Andrew Weaver  
>> wrote:
>>> I'm late to the discussion here, but I want to add my experience from 
>>> dealing with audit logs specifically. 
>>> 
>>> Chronicle has some advantages (binary, compact) but it has a serious 
>>> disadvantage from a consumption standpoint. It's not a well-supported file 
>>> format. Audit logs are something that I think most operators are interested 
>>> in archiving for compliance purposes and analyzing offline for any number 
>>> of reasons and an oddball file format is an unnecessary hurdle for the 
>>> audit logs use-case. 
>>> 
>>> I would welcome support for an existing format that is compact, 
>>> high-performance and compatible with common tools (Spark, etc.).
>>> 
>>> On Sun, Sep 29, 2024, 10:11 AM Štefan Miklošovič  
>>> wrote:
 Thank you all for your answers and opinions. I would like to have some 
 kind of a resolution here in order to move forward, especially with 
 relation to CEP-12 I mentioned earlier. (1).
 
 I think we have these options:
 
 1) Do nothing and wait until this gets back to us, probably in a more 
 serious way (we find a bug and we will not be able to update it because it 
 would be on "ea" or new features will be available only in newer versions).
 
 2) Fork it and continue to maintain it - I do not think this is realistic, 
 nobody is going to t

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

That is all OK to mention. So as I read it correctly, Jon, myself, Andrew,
David - we all would like to see some cross-language events ingestion.
Other people do not seem to consider it important enough (just correct me
if I am wrong). I do not mind, I am 50:50, not an absolute must but sure,
let's add that to the wish list.

I believe that unless it is an absolute show-stopper (which it is not) it
is not necessary. It is something to keep in mind upon next
refactorization. I am personally not affected by this. Whoever will need
that, they will find a way to make it happen.

I would really appreciate it if we found consensus among these points I
wrote down. From my point of view, the most ideal outcome would be to make
CEP-12 happen which cleans it all up and makes it more robust.

My perception is that we have always found the most practical solution and
if it brings value to a user (being able to inspect diagnostic events
offline for further inspection) we should not avoid delivering that.
Something similar happened in e.g. Password validator / generator (CEP-24)
where the most ideal solution was to base it on transactional configuration
but even though we are not there yet, that did not stop us from delivering
it because for some entities in this space it brought value anyway.

It may seem as if I am invested in delivering it because I spent some time
on that already - that is not the case - I am OK to drop diagnostic events
persistence if you insist on that but honestly I just do not see any
compelling argument to do so. The library we are using that for (Chronicle
Queues) is already there, it is just a functionality addition and as I
said, if somebody comes and delivers a solution which would replace it, I
do not see it to be problematic. It would be replaced as anything else
(FQL, Audit ...).

On Sun, Sep 29, 2024 at 9:06 PM Jon Haddad  wrote:

> Strong +1 to the file format issue, and if we're building a wish list - it
> would be great if we could read the file format without pulling in
> cassandra-all.  Long term, I'd love to see this for SSTables & Commit logs
> as well.
>
> I've long been a fan of Gradle subprojects because it makes this kind of
> thing fairly easy.
>
> Jon
>
>
>
> On Sun, Sep 29, 2024 at 11:46 AM Andrew Weaver 
> wrote:
>
>> I'm late to the discussion here, but I want to add my experience from
>> dealing with audit logs specifically.
>>
>> Chronicle has some advantages (binary, compact) but it has a serious
>> disadvantage from a consumption standpoint. It's not a well-supported file
>> format. Audit logs are something that I think most operators are interested
>> in archiving for compliance purposes and analyzing offline for any number
>> of reasons and an oddball file format is an unnecessary hurdle for the
>> audit logs use-case.
>>
>> I would welcome support for an existing format that is compact,
>> high-performance and compatible with common tools (Spark, etc.).
>>
>> On Sun, Sep 29, 2024, 10:11 AM Štefan Miklošovič 
>> wrote:
>>
>>> Thank you all for your answers and opinions. I would like to have some
>>> kind of a resolution here in order to move forward, especially with
>>> relation to CEP-12 I mentioned earlier. (1).
>>>
>>> I think we have these options:
>>>
>>> 1) Do nothing and wait until this gets back to us, probably in a more
>>> serious way (we find a bug and we will not be able to update it because it
>>> would be on "ea" or new features will be available only in newer versions).
>>>
>>> 2) Fork it and continue to maintain it - I do not think this is
>>> realistic, nobody is going to take care of forking that and maintaining it
>>> long term.
>>>
>>> 3) Do nothing but refactor it in such a way that it will be easier to
>>> replace it with something else in the future. CEP-12 is not only adding
>>> persistence to diagnostic events but the patch I have also makes whole
>>> logging more robust. Even it is all on Chronicle Queues (FQL, Audit ...),
>>> there are some differences between that when it comes to the implementation
>>> and I think that refactoring it in such a way that it would have all clear
>>> class structure and hierarchy (bottom of CEP-12) we will have easier job if
>>> we ever go to replace that.
>>>
>>> 4) Proceed with CEP-12 even though we know we are building it on top of
>>> something which should not be there.
>>>
>>> 5) Do absolutely nothing until we replace it with something else and we
>>> get rid of what is there right now - that would mean that we will not
>>> benefit from the code which is easier to maintain etc (if CEP-12 is not
>>> going to materialize) which I think is a welcomed attribute of the code
>>> base to have.
>>>
>>> I was thinking more about stuff like protobuf and while I do see
>>> benefits of that, honestly, it just does not matter too much if it is done
>>> like that or not. I mean, sure, it would be cool to have, but we could
>>> spend a lot of effort on protobuf and integrating with it or on anything
>>> which would

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-29 Thread Jon Haddad

Strong +1 to the file format issue, and if we're building a wish list - it
would be great if we could read the file format without pulling in
cassandra-all.  Long term, I'd love to see this for SSTables & Commit logs
as well.

I've long been a fan of Gradle subprojects because it makes this kind of
thing fairly easy.

Jon



On Sun, Sep 29, 2024 at 11:46 AM Andrew Weaver 
wrote:

> I'm late to the discussion here, but I want to add my experience from
> dealing with audit logs specifically.
>
> Chronicle has some advantages (binary, compact) but it has a serious
> disadvantage from a consumption standpoint. It's not a well-supported file
> format. Audit logs are something that I think most operators are interested
> in archiving for compliance purposes and analyzing offline for any number
> of reasons and an oddball file format is an unnecessary hurdle for the
> audit logs use-case.
>
> I would welcome support for an existing format that is compact,
> high-performance and compatible with common tools (Spark, etc.).
>
> On Sun, Sep 29, 2024, 10:11 AM Štefan Miklošovič 
> wrote:
>
>> Thank you all for your answers and opinions. I would like to have some
>> kind of a resolution here in order to move forward, especially with
>> relation to CEP-12 I mentioned earlier. (1).
>>
>> I think we have these options:
>>
>> 1) Do nothing and wait until this gets back to us, probably in a more
>> serious way (we find a bug and we will not be able to update it because it
>> would be on "ea" or new features will be available only in newer versions).
>>
>> 2) Fork it and continue to maintain it - I do not think this is
>> realistic, nobody is going to take care of forking that and maintaining it
>> long term.
>>
>> 3) Do nothing but refactor it in such a way that it will be easier to
>> replace it with something else in the future. CEP-12 is not only adding
>> persistence to diagnostic events but the patch I have also makes whole
>> logging more robust. Even it is all on Chronicle Queues (FQL, Audit ...),
>> there are some differences between that when it comes to the implementation
>> and I think that refactoring it in such a way that it would have all clear
>> class structure and hierarchy (bottom of CEP-12) we will have easier job if
>> we ever go to replace that.
>>
>> 4) Proceed with CEP-12 even though we know we are building it on top of
>> something which should not be there.
>>
>> 5) Do absolutely nothing until we replace it with something else and we
>> get rid of what is there right now - that would mean that we will not
>> benefit from the code which is easier to maintain etc (if CEP-12 is not
>> going to materialize) which I think is a welcomed attribute of the code
>> base to have.
>>
>> I was thinking more about stuff like protobuf and while I do see benefits
>> of that, honestly, it just does not matter too much if it is done like that
>> or not. I mean, sure, it would be cool to have, but we could spend a lot of
>> effort on protobuf and integrating with it or on anything which would make
>> the consumption of these events language-agnostic but these are quite niche
>> scenarios and I think that time might be used somewhere else more
>> effectively.
>>
>> The bottom line is that I am reluctant to do anything unless CEP-12 makes
>> it in one way or another (either with diagnostic persistence or without it
>> but with a nice refactoring) and, let's get real here, I do not think that
>> anybody is going to spend any time on this particular piece of the
>> functionality either. So the net result is that it will be either
>> athrophying or we at least clean it up so whoever comes next has an easier
>> job to replace it.
>>
>> (1)
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-12+Diagnostics+events+persistence+and+their+exposure+in+virtual+tables
>>
>> On Tue, Sep 24, 2024 at 6:11 PM Ariel Weisberg  wrote:
>>
>>> Hi,
>>>
>>> I just don't understand what "good enough performance" is.
>>>
>>> Should really specify throughput. There is a single thread writing
>>> records to the log and it's a bottleneck around a few hundred thousand
>>> entries/sec and 1gb/sec. It doesn't scale to arbitrary throughput
>>> requirements.
>>>
>>> What is a "predictable footprint"? Was that measured too? How did we
>>> quantify that?
>>>
>>> You can set a rolling cycle to limit the size of the log. It's not that
>>> predictable disk space wise because rolling is time based, and that is one
>>> of the things I don't like about Chronicle.
>>>
>>> This is interesting, if I understand correctly, the messages are
>>> weighted and the heavier they are, the more probable it is they will be
>>> dropped when it is overloaded? Or vice versa, the tighter ones are dropped
>>> first?
>>>
>>> It's still a FIFO queue. Elements aren't dropped from the queue they are
>>> dropped by the producers who don't have to wait for the consumer of the
>>> queue to catch up. The queue size is described in terms of weight not
>>> number of elements so i

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-29 Thread Andrew Weaver

I'm late to the discussion here, but I want to add my experience from
dealing with audit logs specifically.

Chronicle has some advantages (binary, compact) but it has a serious
disadvantage from a consumption standpoint. It's not a well-supported file
format. Audit logs are something that I think most operators are interested
in archiving for compliance purposes and analyzing offline for any number
of reasons and an oddball file format is an unnecessary hurdle for the
audit logs use-case.

I would welcome support for an existing format that is compact,
high-performance and compatible with common tools (Spark, etc.).

On Sun, Sep 29, 2024, 10:11 AM Štefan Miklošovič 
wrote:

> Thank you all for your answers and opinions. I would like to have some
> kind of a resolution here in order to move forward, especially with
> relation to CEP-12 I mentioned earlier. (1).
>
> I think we have these options:
>
> 1) Do nothing and wait until this gets back to us, probably in a more
> serious way (we find a bug and we will not be able to update it because it
> would be on "ea" or new features will be available only in newer versions).
>
> 2) Fork it and continue to maintain it - I do not think this is realistic,
> nobody is going to take care of forking that and maintaining it long term.
>
> 3) Do nothing but refactor it in such a way that it will be easier to
> replace it with something else in the future. CEP-12 is not only adding
> persistence to diagnostic events but the patch I have also makes whole
> logging more robust. Even it is all on Chronicle Queues (FQL, Audit ...),
> there are some differences between that when it comes to the implementation
> and I think that refactoring it in such a way that it would have all clear
> class structure and hierarchy (bottom of CEP-12) we will have easier job if
> we ever go to replace that.
>
> 4) Proceed with CEP-12 even though we know we are building it on top of
> something which should not be there.
>
> 5) Do absolutely nothing until we replace it with something else and we
> get rid of what is there right now - that would mean that we will not
> benefit from the code which is easier to maintain etc (if CEP-12 is not
> going to materialize) which I think is a welcomed attribute of the code
> base to have.
>
> I was thinking more about stuff like protobuf and while I do see benefits
> of that, honestly, it just does not matter too much if it is done like that
> or not. I mean, sure, it would be cool to have, but we could spend a lot of
> effort on protobuf and integrating with it or on anything which would make
> the consumption of these events language-agnostic but these are quite niche
> scenarios and I think that time might be used somewhere else more
> effectively.
>
> The bottom line is that I am reluctant to do anything unless CEP-12 makes
> it in one way or another (either with diagnostic persistence or without it
> but with a nice refactoring) and, let's get real here, I do not think that
> anybody is going to spend any time on this particular piece of the
> functionality either. So the net result is that it will be either
> athrophying or we at least clean it up so whoever comes next has an easier
> job to replace it.
>
> (1)
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-12+Diagnostics+events+persistence+and+their+exposure+in+virtual+tables
>
> On Tue, Sep 24, 2024 at 6:11 PM Ariel Weisberg  wrote:
>
>> Hi,
>>
>> I just don't understand what "good enough performance" is.
>>
>> Should really specify throughput. There is a single thread writing
>> records to the log and it's a bottleneck around a few hundred thousand
>> entries/sec and 1gb/sec. It doesn't scale to arbitrary throughput
>> requirements.
>>
>> What is a "predictable footprint"? Was that measured too? How did we
>> quantify that?
>>
>> You can set a rolling cycle to limit the size of the log. It's not that
>> predictable disk space wise because rolling is time based, and that is one
>> of the things I don't like about Chronicle.
>>
>> This is interesting, if I understand correctly, the messages are weighted
>> and the heavier they are, the more probable it is they will be dropped when
>> it is overloaded? Or vice versa, the tighter ones are dropped first?
>>
>> It's still a FIFO queue. Elements aren't dropped from the queue they are
>> dropped by the producers who don't have to wait for the consumer of the
>> queue to catch up. The queue size is described in terms of weight not
>> number of elements so it can bound memory usage.
>>
>> Have we _ever_ experienced in production that some log events were really
>> dropped? Has anybody ever hit that?
>>
>> Dropping samples is off by default so it can be used in a lossless way.
>>
>> Notionally one of the use cases of full query logging is that you have a
>> cluster that is overloaded and want to find out what is causing it. These
>> nodes maybe low on IO/CPU and turning on the full query log could cause
>> additional timeouts so one goal of th

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-29 Thread Štefan Miklošovič

Thank you all for your answers and opinions. I would like to have some kind
of a resolution here in order to move forward, especially with relation to
CEP-12 I mentioned earlier. (1).

I think we have these options:

1) Do nothing and wait until this gets back to us, probably in a more
serious way (we find a bug and we will not be able to update it because it
would be on "ea" or new features will be available only in newer versions).

2) Fork it and continue to maintain it - I do not think this is realistic,
nobody is going to take care of forking that and maintaining it long term.

3) Do nothing but refactor it in such a way that it will be easier to
replace it with something else in the future. CEP-12 is not only adding
persistence to diagnostic events but the patch I have also makes whole
logging more robust. Even it is all on Chronicle Queues (FQL, Audit ...),
there are some differences between that when it comes to the implementation
and I think that refactoring it in such a way that it would have all clear
class structure and hierarchy (bottom of CEP-12) we will have easier job if
we ever go to replace that.

4) Proceed with CEP-12 even though we know we are building it on top of
something which should not be there.

5) Do absolutely nothing until we replace it with something else and we get
rid of what is there right now - that would mean that we will not benefit
from the code which is easier to maintain etc (if CEP-12 is not going to
materialize) which I think is a welcomed attribute of the code base to have.

I was thinking more about stuff like protobuf and while I do see benefits
of that, honestly, it just does not matter too much if it is done like that
or not. I mean, sure, it would be cool to have, but we could spend a lot of
effort on protobuf and integrating with it or on anything which would make
the consumption of these events language-agnostic but these are quite niche
scenarios and I think that time might be used somewhere else more
effectively.

The bottom line is that I am reluctant to do anything unless CEP-12 makes
it in one way or another (either with diagnostic persistence or without it
but with a nice refactoring) and, let's get real here, I do not think that
anybody is going to spend any time on this particular piece of the
functionality either. So the net result is that it will be either
athrophying or we at least clean it up so whoever comes next has an easier
job to replace it.

(1)
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-12+Diagnostics+events+persistence+and+their+exposure+in+virtual+tables

On Tue, Sep 24, 2024 at 6:11 PM Ariel Weisberg  wrote:

> Hi,
>
> I just don't understand what "good enough performance" is.
>
> Should really specify throughput. There is a single thread writing records
> to the log and it's a bottleneck around a few hundred thousand entries/sec
> and 1gb/sec. It doesn't scale to arbitrary throughput requirements.
>
> What is a "predictable footprint"? Was that measured too? How did we
> quantify that?
>
> You can set a rolling cycle to limit the size of the log. It's not that
> predictable disk space wise because rolling is time based, and that is one
> of the things I don't like about Chronicle.
>
> This is interesting, if I understand correctly, the messages are weighted
> and the heavier they are, the more probable it is they will be dropped when
> it is overloaded? Or vice versa, the tighter ones are dropped first?
>
> It's still a FIFO queue. Elements aren't dropped from the queue they are
> dropped by the producers who don't have to wait for the consumer of the
> queue to catch up. The queue size is described in terms of weight not
> number of elements so it can bound memory usage.
>
> Have we _ever_ experienced in production that some log events were really
> dropped? Has anybody ever hit that?
>
> Dropping samples is off by default so it can be used in a lossless way.
>
> Notionally one of the use cases of full query logging is that you have a
> cluster that is overloaded and want to find out what is causing it. These
> nodes maybe low on IO/CPU and turning on the full query log could cause
> additional timeouts so one goal of the full query log is that enabling it
> shouldn't make things worse.
>
> That is the motivation for memory limits and not blocking request threads
> on IO. Really there should also be rate limits and random sampling because
> right now dropping samples will be biased towards dropping large footprint
> samples.
>
> David Capwell mentioned some performance issues. I recall we talked about
> it and I did a quick microbenchmark and didn't have a problem writing
> records (1 gigabyte/sec, hundreds of thousands of entries) so I am not sure
> what scenarios is where performance is bad and whether it is addressable.
> Not sure it matters since Chronicle's approach to OSS is so problematic.
>
> Ariel
>
> On Tue, Sep 17, 2024, at 4:27 AM, Štefan Miklošovič wrote:
>
> to Benedict:
>
> well ... I was not around whe

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-24 Thread Ariel Weisberg

Hi,

> I just don't understand what "good enough performance" is. 
Should really specify throughput. There is a single thread writing records to 
the log and it's a bottleneck around a few hundred thousand entries/sec and 
1gb/sec. It doesn't scale to arbitrary throughput requirements.

> What is a "predictable footprint"? Was that measured too? How did we quantify 
> that?
You can set a rolling cycle to limit the size of the log. It's not that 
predictable disk space wise because rolling is time based, and that is one of 
the things I don't like about Chronicle.

> This is interesting, if I understand correctly, the messages are weighted and 
> the heavier they are, the more probable it is they will be dropped when it is 
> overloaded? Or vice versa, the tighter ones are dropped first?
It's still a FIFO queue. Elements aren't dropped from the queue they are 
dropped by the producers who don't have to wait for the consumer of the queue 
to catch up. The queue size is described in terms of weight not number of 
elements so it can bound memory usage.

> Have we _ever_ experienced in production that some log events were really 
> dropped? Has anybody ever hit that?
Dropping samples is off by default so it can be used in a lossless way.

Notionally one of the use cases of full query logging is that you have a 
cluster that is overloaded and want to find out what is causing it. These nodes 
maybe low on IO/CPU and turning on the full query log could cause additional 
timeouts so one goal of the full query log is that enabling it shouldn't make 
things worse.

That is the motivation for memory limits and not blocking request threads on 
IO. Really there should also be rate limits and random sampling because right 
now dropping samples will be biased towards dropping large footprint samples.

David Capwell mentioned some performance issues. I recall we talked about it 
and I did a quick microbenchmark and didn't have a problem writing records (1 
gigabyte/sec, hundreds of thousands of entries) so I am not sure what scenarios 
is where performance is bad and whether it is addressable. Not sure it matters 
since Chronicle's approach to OSS is so problematic. 

Ariel

On Tue, Sep 17, 2024, at 4:27 AM, Štefan Miklošovič wrote:
> to Benedict:
> 
> well ... I was not around when the decision about the usage of Chronicle 
> Queues was made. I think that at that time it was the most obvious candidate 
> without reinventing the wheel given the features and capabilities it had so 
> taking something off the shelf was a natural conclusion.
> 
> Josh / Jordan:
> 
> not only FQL but Audit as well these are two separate things. There is also 
> quite a "rich" ecosystem around that.
> 
> 1) nodetool commands like
> 
> enableauditlog
> enablefullquerylog
> disableauditlog
> disablefullquerylog
> getauditlog
> getfullquerylog
> 
> Also, because the files it produces are binary, we need a special tooling to 
> inspect it, it is in tools/fqltool with a bunch of classes, and there is also 
> an AuditLogViewer for reviewing audit logs.
> 
> There are MBean methods enabling nodetool commands.
> 
> We have also shipped that in two major releases (4.0 and now in 5.0) so the 
> community is quite well used to this, they have the processes set around this 
> etc.
> 
> I mention this all because it is just not so easy to replace it with 
> something else if somebody wanted that, in any case. How do we even go around 
> deprecating this if we are indeed going to replace that?
> 
> To discuss the release aspect they have in place: I think you are right that 
> the latest ea is as close as possible, if not the same, as what they release 
> privately. Yes. But if we want to stick to the rule that we upgrade only to 
> the latest ea relese before their next minor, then 
> 
> 1) we will be always at least one minor late
> 2) we do not know when they make up their minds to transition to a new minor 
> so we can upgrade to the latest ea one minor before 
> 3) if something is broken and we need to fix it and we are on ea, then what 
> we get to update to is the latest ea at that time which might fix the issue 
> but it will also bring new stuff in which might open doors to instability as 
> well. So we update to fix the bugs but we might include new ones unknowingly.
> 
> Anyway, I don't think this has any silver bullet solution, we might just 
> stick to the latest "ea" and be done with it. I do not expect this project to 
> evolve wildly and unpredictably, it just solves "one problem", there is 
> basically nothing new coming in.
> 
> Brandon:
> 
> I understand your concerns about phoning home but 
> 
> 1) we already resolved this by setting the respective property
> 2) I do not think that Chronicle will mess with this once they introduce 
> that. There is nothing to "improve" or "change" there. It is phoning home or 
> not and it is driven by one property. If they made a change that we can not 
> turn it off then we would really be in trouble bu

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-24 Thread Ariel Weisberg

Hi,

A bit late to this discussion, there are few emails in the thread I'll respond 
to separately.

Yes it's just to implement a binary file based queue that supports realtime 
tailing. If it had been low friction it would have made more sense, but given 
the friction Chronicle's approach to OSS brings I don't think it makes much 
sense to keep using it.

It has some jankiness to it. I vaguely recall having copied the files while 
they are being written to and then gotten errors trying to read them.

I think the next time we want to add functionality to the FQL/binlog it would 
be worth re-implementing that part or even just doing it to get rid of the 
dependency.

Ariel

On Mon, Sep 16, 2024, at 5:16 PM, Benedict wrote:
> 
> Don’t we essentially just use it as a file format for storing a couple of 
> kinds of append-only data?
> 
> I was never entirely clear on the value it brought to the project.
> 
> 
>> On 16 Sep 2024, at 22:11, Jordan West  wrote:
>> 
>> Thanks for the sleuthing Stefan! This definitely is a bit unfortunate. It 
>> sounds like a replacement is not really practical so I'll ignore that option 
>> for now, until a viable alternative is proposed. I am -1 on us writing our 
>> own without strong, strong justification -- primarily because I think the 
>> likelihood is we introduce more bugs before getting to something stable. 
>> 
>> Regarding the remaining options, mostly some thoughts:
>> 
>> - it would be nice to have some specific evidence of other projects using 
>> the EA versions and what their developers have said about it.
>> - it sounds like if we go with the EA route, the onus to test for 
>> correctness / compatibility increases. They do test but anything marked 
>> "early access" I think deserves more scrutiny from the C* community before 
>> release. That could come in the form of more tests (or showing that we 
>> already have good coverage of where its used).
>> - i assume each time we upgrade we would pick the most recently released EA 
>> version
>> 
>> Jordan
>> 
>> 
>> On Mon, Sep 16, 2024 at 1:46 PM Štefan Miklošovič  
>> wrote:
>>> We are using a library called Chronicle Queue (1) and its dependencies and 
>>> we ship them in the distribution tarball.
>>> 
>>> The version we use in 5.0 / trunk as I write this is 2.23.36. If you look 
>>> closely here (2), there is one more release like this, 2.23.37 and after 
>>> that all these releases have "ea" in their name.
>>> 
>>> "ea" stands for "early access". The project has changed the versioning / 
>>> development model in such a way that "ea" releases act, more or less, as 
>>> glorified snapshots which are indeed released to Maven Central but the 
>>> "regular" releases are not there. The reason behind this is that "regular" 
>>> releases are published only for customers who pay to the company behind 
>>> this project and they offer commercial support for that.
>>> 
>>> "regular" releases are meant to get all the bug fixes after "ea" is 
>>> published and they are official stable releases. On the other hand "ea" 
>>> releases are the ones where the development happens and every now and then, 
>>> once the developers think that it is time to cut new 2.x, they just publish 
>>> that privately.
>>> 
>>> I was investigating how this all works here (3) and while they said that, I 
>>> quote (4):
>>> 
>>> "In my experience this is consumed by a large number of open source 
>>> projects reliably (for our other artifacts too). This development/ea branch 
>>> still goes through an extensive test suite prior to release. Releases from 
>>> this branch will contain the latest features and bug fixes."
>>> 
>>> I am not completely sure if we are OK with this. For the record, Mick is 
>>> not overly comfortable with that and Brandon would prefer to just replace 
>>> it / get rid of this dependency (comments / reasons / discussion from (5) 
>>> to the end)
>>> 
>>> The question is if we are OK with how things are and if we are then what 
>>> are the rules when upgrading the version of this project in Cassandra in 
>>> the context of "ea" versions they publish.
>>> 
>>> If we are not OK with this, then the question is what we are going to 
>>> replace it with.
>>> 
>>> If we are going to replace it, I very briefly took a look and there is 
>>> practically nothing out there which would hit all the buttons for us. 
>>> Chronicle is just perfect for this job and I am not a fan of rewriting this 
>>> at all. 
>>> 
>>> I would like to have this resolved because there is CEP-12 I plan to 
>>> deliver and I hit this and I do not want to base that work on something we 
>>> might eventually abandon. There are some ideas for CEP-12 how to bypass 
>>> this without using Chronicle but I would like to firstly hear your opinion.
>>> 
>>> Regards
>>> 
>>> (1) https://github.com/OpenHFT/Chronicle-Queue
>>> (2) https://repo1.maven.org/maven2/net/openhft/chronicle-core/
>>> (3) https://github.com/OpenHFT/Chronicle-Core/issues/668
>>> (4) 
>>> https://gith

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread David Capwell

> I think reserializing the payload into a new format is counter productive to 
> some of the performance goals of the binary logs? If you have to deserialize 
> and reserialize the message you are going to be throwing off a ton of extra 
> GC.

That isn’t what happens in FQL =D.

FQL creates a custom payload using Chronicle fields, then serializes 
QueryOptions (we have in-memory objects we use for the query).

We are not taking the client network bytes and saving to a log (client bytes 
could be different pages… this would be annoying to support), we are working 
with the following

String query
ByteBuffer[] binds
QueryOptions options

“Could” we use our networking serializer?  Sure, but then what do we get?  The 
cost to construct the object to pass to the serializer is basically the same, 
so it’s just the time it takes to serialize it, and I argue in this very 
specific case the costs are not really that noticeable (and have 
benchmarked...).

So, we put a burden on users (and us to maintain binary compatibility with 
QueryOptions), making it harder for them at the cost of a few nanoseconds more 
to serialize?

> On Sep 19, 2024, at 3:32 PM, Štefan Miklošovič  wrote:
> 
> Wow this is quite a rabbit hole. 
> 
> What is ultimately going to be written into Chronicle Queue is what 
> writeMarshallablePayload method on AbstractLogQuery puts into that WireOut. 
> If we take e.g. QUERY_OPTIONS into consideration, then it writes it into 
> queryOptionsBuffer which is populated in AbstractLogEntry's constructor 
> (QueryOptions.codec.encode).
> 
> That takes QueryOptions, deserialized stuff we got from QueryMessage via 
> codec in its decode method, and it encodes it back to ByteBuf. So for now, we 
> just serialize what we deserialized all over again. 
> 
> But for what reason do we need to serialize it again upon logging it to FQL? 
> "body" here which is used for decoding bytes to QueryOptions is ByteBuf 
> already. So if we go to write bytes then well here we have it. It does not 
> seem to be necessary to decode / encode, just use these bytes as they are?
> 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/transport/messages/QueryMessage.java#L51
> 
> On Thu, Sep 19, 2024 at 11:50 PM Benedict Elliott Smith  > wrote:
>> Well, that looks like item number one to fix when we change the 
>> serialisation format. We should clearly not duplicate query strings we have 
>> recently logged.
>> 
>> We do however appear to also serialise the bind variables, which benefit 
>> from being in the format we already have available in memory.
>> 
>>> On 19 Sep 2024, at 22:26, Štefan Miklošovič >> > wrote:
>>> 
>>> I am not sure what you mean. I mean, I do, but not following. Look into 
>>> FullQueryLogger (1) what it goes to put into CQL is a query like String 
>>> wrapped in a Query object. It literally take a String as a representation 
>>> of a query a user executed. We just replace this by serializing that query 
>>> to protobuf. What is counter productive? We just replace one thing for 
>>> another. Audit message / events would be similar. 
>>> 
>>> (1) 
>>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/fql/FullQueryLogger.java#L320-L321
>>> 
>>> On Thu, Sep 19, 2024 at 11:17 PM J. D. Jordan >> > wrote:
 I think reserializing the payload into a new format is counter productive 
 to some of the performance goals of the binary logs?
 If you have to deserialize and reserialize the message you are going to be 
 throwing off a ton of extra GC.
 I think we have done a lot of work in recent version to reduce the amount 
 of re-serialization that happens in the query paths?  Not sure we want to 
 add some back in on purpose?  Keeping the payload in the internal 
 serialization format does indeed have the drawbacks David mentioned, but I 
 think “zero serialization overhead” is a pretty big advantage to keeping 
 things that way?
 
 -Jeremiah
 
> On Sep 19, 2024, at 3:56 PM, Štefan Miklošovič  > wrote:
> 
> 
> I think protobuf upon serialization is just a bunch of bytes anyway. If 
> we figure out the header as David showed then we can still serialize it 
> all with the machinery / serializers you mentioned. It can write bytes, 
> right?! I very briefly checked and I think that protobuf is super simple 
> and does not have any checksumming etc. so some sauce on top of that 
> would be necessary anyway and we can reuse what we have to produce binary 
> files.
> 
> On the consumer side, the binary file would be parsed with some tooling 
> e.g. in Go, indeed, but the headers and stuff would be so simple that it 
> would be just a coding exercise and then it might be deserialized with 
> protobuf for that language.
> 
> Basical

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

Wow this is quite a rabbit hole.

What is ultimately going to be written into Chronicle Queue is what
writeMarshallablePayload method on AbstractLogQuery puts into that WireOut.
If we take e.g. QUERY_OPTIONS into consideration, then it writes it into
queryOptionsBuffer which is populated in AbstractLogEntry's constructor
(QueryOptions.codec.encode).

That takes QueryOptions, deserialized stuff we got from QueryMessage via
codec in its decode method, and it encodes it back to ByteBuf. So for now,
we just serialize what we deserialized all over again.

But for what reason do we need to serialize it again upon logging it to
FQL? "body" here which is used for decoding bytes to QueryOptions is
ByteBuf already. So if we go to write bytes then well here we have it. It
does not seem to be necessary to decode / encode, just use these bytes as
they are?

https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/transport/messages/QueryMessage.java#L51

On Thu, Sep 19, 2024 at 11:50 PM Benedict Elliott Smith 
wrote:

> Well, that looks like item number one to fix when we change the
> serialisation format. We should clearly not duplicate query strings we have
> recently logged.
>
> We do however appear to also serialise the bind variables, which benefit
> from being in the format we already have available in memory.
>
> On 19 Sep 2024, at 22:26, Štefan Miklošovič 
> wrote:
>
> I am not sure what you mean. I mean, I do, but not following. Look into
> FullQueryLogger (1) what it goes to put into CQL is a query like String
> wrapped in a Query object. It literally take a String as a representation
> of a query a user executed. We just replace this by serializing that query
> to protobuf. What is counter productive? We just replace one thing for
> another. Audit message / events would be similar.
>
> (1)
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/fql/FullQueryLogger.java#L320-L321
>
> On Thu, Sep 19, 2024 at 11:17 PM J. D. Jordan 
> wrote:
>
>> I think reserializing the payload into a new format is counter productive
>> to some of the performance goals of the binary logs?
>> If you have to deserialize and reserialize the message you are going to
>> be throwing off a ton of extra GC.
>> I think we have done a lot of work in recent version to reduce the amount
>> of re-serialization that happens in the query paths?  Not sure we want to
>> add some back in on purpose?  Keeping the payload in the internal
>> serialization format does indeed have the drawbacks David mentioned, but I
>> think “zero serialization overhead” is a pretty big advantage to keeping
>> things that way?
>>
>> -Jeremiah
>>
>> On Sep 19, 2024, at 3:56 PM, Štefan Miklošovič 
>> wrote:
>>
>> 
>> I think protobuf upon serialization is just a bunch of bytes anyway. If
>> we figure out the header as David showed then we can still serialize it all
>> with the machinery / serializers you mentioned. It can write bytes, right?!
>> I very briefly checked and I think that protobuf is super simple and does
>> not have any checksumming etc. so some sauce on top of that would be
>> necessary anyway and we can reuse what we have to produce binary files.
>>
>> On the consumer side, the binary file would be parsed with some tooling
>> e.g. in Go, indeed, but the headers and stuff would be so simple that it
>> would be just a coding exercise and then it might be deserialized with
>> protobuf for that language.
>>
>> Basically, only the payload itself would be the product of protobuf and
>> all around super simple to crack through.
>>
>> On Thu, Sep 19, 2024 at 10:41 PM Benedict  wrote:
>>
>>> Sorry, I missed that. I’m not convinced any of these logs need language
>>> agnostic tools for access, but if that’s a goal for other folk I don’t feel
>>> strongly about it.
>>>
>>> On 19 Sep 2024, at 21:06, Štefan Miklošovič 
>>> wrote:
>>>
>>> 
>>> More to it, it is actually not only about FQL. Audit logging is on
>>> Chronicle queues too so inspecting that would be platform independent as
>>> well.
>>>
>>> CEP-12 suggests that there might be a persistent store for diagnostic
>>> events as well. If somebody wants to inspect what a node was doing after it
>>> went offline as for now all these events are in memory only.
>>>
>>> This would basically enable people to fully inspect what the cluster was
>>> doing from FQL to Audit to Diagnostics in a language independent manner.
>>>
>>> On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič <
>>> [email protected]> wrote:
>>>
 I think the biggest selling point for using something like protobuf is
 what David said - what if he wants to replay it in Go? Basing it on
 something language neutral enables people to replay it in whatever they
 want. If we have something totally custom then it is replayable just in
 Java without bringing tons of dependencies to their projects. That is the
 message I got from what he wrote.

 On Thu, Sep 19, 2024 at 9:47 PM B

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread Benedict Elliott Smith

Well, that looks like item number one to fix when we change the serialisation 
format. We should clearly not duplicate query strings we have recently logged.

We do however appear to also serialise the bind variables, which benefit from 
being in the format we already have available in memory.

> On 19 Sep 2024, at 22:26, Štefan Miklošovič  wrote:
> 
> I am not sure what you mean. I mean, I do, but not following. Look into 
> FullQueryLogger (1) what it goes to put into CQL is a query like String 
> wrapped in a Query object. It literally take a String as a representation of 
> a query a user executed. We just replace this by serializing that query to 
> protobuf. What is counter productive? We just replace one thing for another. 
> Audit message / events would be similar. 
> 
> (1) 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/fql/FullQueryLogger.java#L320-L321
> 
> On Thu, Sep 19, 2024 at 11:17 PM J. D. Jordan  > wrote:
>> I think reserializing the payload into a new format is counter productive to 
>> some of the performance goals of the binary logs?
>> If you have to deserialize and reserialize the message you are going to be 
>> throwing off a ton of extra GC.
>> I think we have done a lot of work in recent version to reduce the amount of 
>> re-serialization that happens in the query paths?  Not sure we want to add 
>> some back in on purpose?  Keeping the payload in the internal serialization 
>> format does indeed have the drawbacks David mentioned, but I think “zero 
>> serialization overhead” is a pretty big advantage to keeping things that way?
>> 
>> -Jeremiah
>> 
>>> On Sep 19, 2024, at 3:56 PM, Štefan Miklošovič >> > wrote:
>>> 
>>> 
>>> I think protobuf upon serialization is just a bunch of bytes anyway. If we 
>>> figure out the header as David showed then we can still serialize it all 
>>> with the machinery / serializers you mentioned. It can write bytes, right?! 
>>> I very briefly checked and I think that protobuf is super simple and does 
>>> not have any checksumming etc. so some sauce on top of that would be 
>>> necessary anyway and we can reuse what we have to produce binary files.
>>> 
>>> On the consumer side, the binary file would be parsed with some tooling 
>>> e.g. in Go, indeed, but the headers and stuff would be so simple that it 
>>> would be just a coding exercise and then it might be deserialized with 
>>> protobuf for that language.
>>> 
>>> Basically, only the payload itself would be the product of protobuf and all 
>>> around super simple to crack through.
>>> 
>>> On Thu, Sep 19, 2024 at 10:41 PM Benedict >> > wrote:
 Sorry, I missed that. I’m not convinced any of these logs need language 
 agnostic tools for access, but if that’s a goal for other folk I don’t 
 feel strongly about it.
 
> On 19 Sep 2024, at 21:06, Štefan Miklošovič  > wrote:
> 
> 
> More to it, it is actually not only about FQL. Audit logging is on 
> Chronicle queues too so inspecting that would be platform independent as 
> well. 
> 
> CEP-12 suggests that there might be a persistent store for diagnostic 
> events as well. If somebody wants to inspect what a node was doing after 
> it went offline as for now all these events are in memory only.
> 
> This would basically enable people to fully inspect what the cluster was 
> doing from FQL to Audit to Diagnostics in a language independent manner. 
> 
> On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič  > wrote:
>> I think the biggest selling point for using something like protobuf is 
>> what David said - what if he wants to replay it in Go? Basing it on 
>> something language neutral enables people to replay it in whatever they 
>> want. If we have something totally custom then it is replayable just in 
>> Java without bringing tons of dependencies to their projects. That is 
>> the message I got from what he wrote. 
>> 
>> On Thu, Sep 19, 2024 at 9:47 PM Benedict > > wrote:
>>> Do we need any of these things either? We have our own serialisation 
>>> framework and file readers and writers, and at least in the FQL case 
>>> these are the native serialisation format. 
>>> 
>>> At cursory glance it also looks to me like this would be a minimal 
>>> refactor from the current state.
>>> 
>>> What is the reason we want to add these other dependencies?
>>> 
>>> 
 On 19 Sep 2024, at 20:31, Štefan Miklošovič >>> > wrote:
 
 
 well the Maven plugin declares that it downloads protoc from Maven 
 Central automatically _somehow_ so coding up an Ant task which does 
 something similar shouldn't be too hard

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I am not sure what you mean. I mean, I do, but not following. Look into
FullQueryLogger (1) what it goes to put into CQL is a query like String
wrapped in a Query object. It literally take a String as a representation
of a query a user executed. We just replace this by serializing that query
to protobuf. What is counter productive? We just replace one thing for
another. Audit message / events would be similar.

(1)
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/fql/FullQueryLogger.java#L320-L321

On Thu, Sep 19, 2024 at 11:17 PM J. D. Jordan 
wrote:

> I think reserializing the payload into a new format is counter productive
> to some of the performance goals of the binary logs?
> If you have to deserialize and reserialize the message you are going to be
> throwing off a ton of extra GC.
> I think we have done a lot of work in recent version to reduce the amount
> of re-serialization that happens in the query paths?  Not sure we want to
> add some back in on purpose?  Keeping the payload in the internal
> serialization format does indeed have the drawbacks David mentioned, but I
> think “zero serialization overhead” is a pretty big advantage to keeping
> things that way?
>
> -Jeremiah
>
> On Sep 19, 2024, at 3:56 PM, Štefan Miklošovič 
> wrote:
>
> 
> I think protobuf upon serialization is just a bunch of bytes anyway. If we
> figure out the header as David showed then we can still serialize it all
> with the machinery / serializers you mentioned. It can write bytes, right?!
> I very briefly checked and I think that protobuf is super simple and does
> not have any checksumming etc. so some sauce on top of that would be
> necessary anyway and we can reuse what we have to produce binary files.
>
> On the consumer side, the binary file would be parsed with some tooling
> e.g. in Go, indeed, but the headers and stuff would be so simple that it
> would be just a coding exercise and then it might be deserialized with
> protobuf for that language.
>
> Basically, only the payload itself would be the product of protobuf and
> all around super simple to crack through.
>
> On Thu, Sep 19, 2024 at 10:41 PM Benedict  wrote:
>
>> Sorry, I missed that. I’m not convinced any of these logs need language
>> agnostic tools for access, but if that’s a goal for other folk I don’t feel
>> strongly about it.
>>
>> On 19 Sep 2024, at 21:06, Štefan Miklošovič 
>> wrote:
>>
>> 
>> More to it, it is actually not only about FQL. Audit logging is on
>> Chronicle queues too so inspecting that would be platform independent as
>> well.
>>
>> CEP-12 suggests that there might be a persistent store for diagnostic
>> events as well. If somebody wants to inspect what a node was doing after it
>> went offline as for now all these events are in memory only.
>>
>> This would basically enable people to fully inspect what the cluster was
>> doing from FQL to Audit to Diagnostics in a language independent manner.
>>
>> On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič 
>> wrote:
>>
>>> I think the biggest selling point for using something like protobuf is
>>> what David said - what if he wants to replay it in Go? Basing it on
>>> something language neutral enables people to replay it in whatever they
>>> want. If we have something totally custom then it is replayable just in
>>> Java without bringing tons of dependencies to their projects. That is the
>>> message I got from what he wrote.
>>>
>>> On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:
>>>
 Do we need any of these things either? We have our own serialisation
 framework and file readers and writers, and at least in the FQL case these
 are the native serialisation format.

 At cursory glance it also looks to me like this would be a minimal
 refactor from the current state.

 What is the reason we want to add these other dependencies?


 On 19 Sep 2024, at 20:31, Štefan Miklošovič 
 wrote:

 
 well the Maven plugin declares that it downloads protoc from Maven
 Central automatically _somehow_ so coding up an Ant task which does
 something similar shouldn't be too hard. I will investigate this idea.

 On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams 
 wrote:

> On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
>  wrote:
> > Unfortunately there is nothing like that for Ant, protoc would need
> to be a local dependency on the computer which compiles the project to be
> able to do that so that is kind of a dead end. Or is there any workaround
> here?
>
> In the old thrift days I believe we generated the code and checked it
> in so you didn't need to compile locally.
>
> Kind Regards,
> Brandon
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I agree, even if we don’t manage the optimal zero conversion. I am also not entirely convinced we need to worry about compatibility for FQL and other logs - we can just say you must use the version of C* tools you produced the log with - I would even be fine with saying this isn’t even guaranteed to be compatible across minor versions (at least for FQL, perhaps not audit logging. But we can and should be explicit about our guarantees each file we produce).Most of this is not meant to be baked into production workflows, and if they are they can use command line tools bundled with C*. Programmatic consumers are primarily going to be us here on this list. Let’s not burden ourselves unnecessarily.On 19 Sep 2024, at 22:17, J. D. Jordan  wrote:I think reserializing the payload into a new format is counter productive to some of the performance goals of the binary logs?If you have to deserialize and reserialize the message you are going to be throwing off a ton of extra GC.I think we have done a lot of work in recent version to reduce the amount of re-serialization that happens in the query paths?  Not sure we want to add some back in on purpose?  Keeping the payload in the internal serialization format does indeed have the drawbacks David mentioned, but I think “zero serialization overhead” is a pretty big advantage to keeping things that way?-JeremiahOn Sep 19, 2024, at 3:56 PM, Štefan Miklošovič  wrote:I think protobuf upon serialization is just a bunch of bytes anyway. If we figure out the header as David showed then we can still serialize it all with the machinery / serializers you mentioned. It can write bytes, right?! I very briefly checked and I think that protobuf is super simple and does not have any checksumming etc. so some sauce on top of that would be necessary anyway and we can reuse what we have to produce binary files.On the consumer side, the binary file would be parsed with some tooling e.g. in Go, indeed, but the headers and stuff would be so simple that it would be just a coding exercise and then it might be deserialized with protobuf for that language.Basically, only the payload itself would be the product of protobuf and all around super simple to crack through.On Thu, Sep 19, 2024 at 10:41 PM Benedict  wrote:Sorry, I missed that. I’m not convinced any of these logs need language agnostic tools for access, but if that’s a goal for other folk I don’t feel strongly about it.On 19 Sep 2024, at 21:06, Štefan Miklošovič  wrote:More to it, it is actually not only about FQL. Audit logging is on Chronicle queues too so inspecting that would be platform independent as well. CEP-12 suggests that there might be a persistent store for diagnostic events as well. If somebody wants to inspect what a node was doing after it went offline as for now all these events are in memory only.This would basically enable people to fully inspect what the cluster was doing from FQL to Audit to Diagnostics in a language independent manner. On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič  wrote:I think the biggest selling point for using something like protobuf is what David said - what if he wants to replay it in Go? Basing it on something language neutral enables people to replay it in whatever they want. If we have something totally custom then it is replayable just in Java without bringing tons of dependencies to their projects. That is the message I got from what he wrote. On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:Do we need any of these things either? We have our own serialisation framework and file readers and writers, and at least in the FQL case these are the native serialisation format. At cursory glance it also looks to me like this would be a minimal refactor from the current state.What is the reason we want to add these other dependencies?On 19 Sep 2024, at 20:31, Štefan Miklošovič  wrote:well the Maven plugin declares that it downloads protoc from Maven Central automatically _somehow_ so coding up an Ant task which does something similar shouldn't be too hard. I will investigate this idea. On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
 wrote:
> Unfortunately there is nothing like that for Ant, protoc would need to be a local dependency on the computer which compiles the project to be able to do that so that is kind of a dead end. Or is there any workaround here?

In the old thrift days I believe we generated the code and checked it
in so you didn't need to compile locally.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread J. D. Jordan

I think reserializing the payload into a new format is counter productive to some of the performance goals of the binary logs?If you have to deserialize and reserialize the message you are going to be throwing off a ton of extra GC.I think we have done a lot of work in recent version to reduce the amount of re-serialization that happens in the query paths?  Not sure we want to add some back in on purpose?  Keeping the payload in the internal serialization format does indeed have the drawbacks David mentioned, but I think “zero serialization overhead” is a pretty big advantage to keeping things that way?-JeremiahOn Sep 19, 2024, at 3:56 PM, Štefan Miklošovič  wrote:I think protobuf upon serialization is just a bunch of bytes anyway. If we figure out the header as David showed then we can still serialize it all with the machinery / serializers you mentioned. It can write bytes, right?! I very briefly checked and I think that protobuf is super simple and does not have any checksumming etc. so some sauce on top of that would be necessary anyway and we can reuse what we have to produce binary files.On the consumer side, the binary file would be parsed with some tooling e.g. in Go, indeed, but the headers and stuff would be so simple that it would be just a coding exercise and then it might be deserialized with protobuf for that language.Basically, only the payload itself would be the product of protobuf and all around super simple to crack through.On Thu, Sep 19, 2024 at 10:41 PM Benedict  wrote:Sorry, I missed that. I’m not convinced any of these logs need language agnostic tools for access, but if that’s a goal for other folk I don’t feel strongly about it.On 19 Sep 2024, at 21:06, Štefan Miklošovič  wrote:More to it, it is actually not only about FQL. Audit logging is on Chronicle queues too so inspecting that would be platform independent as well. CEP-12 suggests that there might be a persistent store for diagnostic events as well. If somebody wants to inspect what a node was doing after it went offline as for now all these events are in memory only.This would basically enable people to fully inspect what the cluster was doing from FQL to Audit to Diagnostics in a language independent manner. On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič  wrote:I think the biggest selling point for using something like protobuf is what David said - what if he wants to replay it in Go? Basing it on something language neutral enables people to replay it in whatever they want. If we have something totally custom then it is replayable just in Java without bringing tons of dependencies to their projects. That is the message I got from what he wrote. On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:Do we need any of these things either? We have our own serialisation framework and file readers and writers, and at least in the FQL case these are the native serialisation format. At cursory glance it also looks to me like this would be a minimal refactor from the current state.What is the reason we want to add these other dependencies?On 19 Sep 2024, at 20:31, Štefan Miklošovič  wrote:well the Maven plugin declares that it downloads protoc from Maven Central automatically _somehow_ so coding up an Ant task which does something similar shouldn't be too hard. I will investigate this idea. On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
 wrote:
> Unfortunately there is nothing like that for Ant, protoc would need to be a local dependency on the computer which compiles the project to be able to do that so that is kind of a dead end. Or is there any workaround here?

In the old thrift days I believe we generated the code and checked it
in so you didn't need to compile locally.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I think protobuf upon serialization is just a bunch of bytes anyway. If we
figure out the header as David showed then we can still serialize it all
with the machinery / serializers you mentioned. It can write bytes, right?!
I very briefly checked and I think that protobuf is super simple and does
not have any checksumming etc. so some sauce on top of that would be
necessary anyway and we can reuse what we have to produce binary files.

On the consumer side, the binary file would be parsed with some tooling
e.g. in Go, indeed, but the headers and stuff would be so simple that it
would be just a coding exercise and then it might be deserialized with
protobuf for that language.

Basically, only the payload itself would be the product of protobuf and all
around super simple to crack through.

On Thu, Sep 19, 2024 at 10:41 PM Benedict  wrote:

> Sorry, I missed that. I’m not convinced any of these logs need language
> agnostic tools for access, but if that’s a goal for other folk I don’t feel
> strongly about it.
>
> On 19 Sep 2024, at 21:06, Štefan Miklošovič 
> wrote:
>
> 
> More to it, it is actually not only about FQL. Audit logging is on
> Chronicle queues too so inspecting that would be platform independent as
> well.
>
> CEP-12 suggests that there might be a persistent store for diagnostic
> events as well. If somebody wants to inspect what a node was doing after it
> went offline as for now all these events are in memory only.
>
> This would basically enable people to fully inspect what the cluster was
> doing from FQL to Audit to Diagnostics in a language independent manner.
>
> On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič 
> wrote:
>
>> I think the biggest selling point for using something like protobuf is
>> what David said - what if he wants to replay it in Go? Basing it on
>> something language neutral enables people to replay it in whatever they
>> want. If we have something totally custom then it is replayable just in
>> Java without bringing tons of dependencies to their projects. That is the
>> message I got from what he wrote.
>>
>> On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:
>>
>>> Do we need any of these things either? We have our own serialisation
>>> framework and file readers and writers, and at least in the FQL case these
>>> are the native serialisation format.
>>>
>>> At cursory glance it also looks to me like this would be a minimal
>>> refactor from the current state.
>>>
>>> What is the reason we want to add these other dependencies?
>>>
>>>
>>> On 19 Sep 2024, at 20:31, Štefan Miklošovič 
>>> wrote:
>>>
>>> 
>>> well the Maven plugin declares that it downloads protoc from Maven
>>> Central automatically _somehow_ so coding up an Ant task which does
>>> something similar shouldn't be too hard. I will investigate this idea.
>>>
>>> On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams 
>>> wrote:
>>>
 On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
  wrote:
 > Unfortunately there is nothing like that for Ant, protoc would need
 to be a local dependency on the computer which compiles the project to be
 able to do that so that is kind of a dead end. Or is there any workaround
 here?

 In the old thrift days I believe we generated the code and checked it
 in so you didn't need to compile locally.

 Kind Regards,
 Brandon

>>>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread David Capwell

> Do we need any of these things either? We have our own serialisation 
> framework and file readers and writers, and at least in the FQL case these 
> are the native serialisation format. 
> 
> At cursory glance it also looks to me like this would be a minimal refactor 
> from the current state.
> 
> What is the reason we want to add these other dependencies?

It’s all about the target user of the feature.  I can’t speak for audit logging 
(why do we need more than slf4j?  No clue), but one of the users for Chronicle 
Queue is FQL.  We do have a FQL reply command line, but we are not trying to 
make this a powerful tool with a ton of ways to replay with different rates, 
interleaving, etc… its a basic “I run w/e I see in the logs”.  We have kinda 
moved away from stress tools being in-tree and letting this evolve outside of 
our code base, so looking at this direction it makes sense for FQL reply to be 
external to the Apache Cassandra tree… in fact leveraging existing tools made 
FQL faster and far more powerful than the in-tree version…

With all that, my view of who the target user of FQL data is the 
users/developers and not internal to Apache Cassandra… so I then need to ask 
what is the experience using it…

If we use internal serializers (that are strongly coupled with our internal 
networking) then the user needs to depend on cassandra-all… this brings in 74 
dependencies (see 
https://mvnrepository.com/artifact/org.apache.cassandra/cassandra-all/5.0.0) 
and non of these matter to the user, so you must exclude every single one or 
you just accept w/e we bring in (which means you are stuck with Java Driver 3, 
can’t use Java Driver 4).

Now that you got that out of the way, you can add the log reading into your 
tools and do what you want, as long as they are in Java.

So, using our “serialization framework” only seems to come with burdens to me

1) its versioning is our internal message versioning, so if we make changes to 
our networking FQL is forced to bump its version as well.  If we need to change 
the log format we need to also bump our networking version…. 
2) for the project we add even more public classes to the list we need to 
maintain compatibility with (I have no clue what is public right now, we debate 
this w/e it comes up), so refactoring our CQL processing layer gets harder
3) cassandra-all is massive
4) in order to reuse outside of java we need to implement translations to a 
more common format so other languages can use… I do have tools in python to 
read the Thrift FQL log I write and compute stats on user behavior… it would be 
nice to leverage the log file directly and not have to translate it

> On Sep 19, 2024, at 1:04 PM, Štefan Miklošovič  wrote:
> 
> More to it, it is actually not only about FQL. Audit logging is on Chronicle 
> queues too so inspecting that would be platform independent as well. 
> 
> CEP-12 suggests that there might be a persistent store for diagnostic events 
> as well. If somebody wants to inspect what a node was doing after it went 
> offline as for now all these events are in memory only.
> 
> This would basically enable people to fully inspect what the cluster was 
> doing from FQL to Audit to Diagnostics in a language independent manner. 
> 
> On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič  > wrote:
>> I think the biggest selling point for using something like protobuf is what 
>> David said - what if he wants to replay it in Go? Basing it on something 
>> language neutral enables people to replay it in whatever they want. If we 
>> have something totally custom then it is replayable just in Java without 
>> bringing tons of dependencies to their projects. That is the message I got 
>> from what he wrote. 
>> 
>> On Thu, Sep 19, 2024 at 9:47 PM Benedict > > wrote:
>>> Do we need any of these things either? We have our own serialisation 
>>> framework and file readers and writers, and at least in the FQL case these 
>>> are the native serialisation format. 
>>> 
>>> At cursory glance it also looks to me like this would be a minimal refactor 
>>> from the current state.
>>> 
>>> What is the reason we want to add these other dependencies?
>>> 
>>> 
 On 19 Sep 2024, at 20:31, Štefan Miklošovič >>> > wrote:

 well the Maven plugin declares that it downloads protoc from Maven Central 
 automatically _somehow_ so coding up an Ant task which does something 
 similar shouldn't be too hard. I will investigate this idea. 

 On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams >>> > wrote:
> On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
> mailto:[email protected]>> wrote:
> > Unfortunately there is nothing like that for Ant, protoc would need to 
> > be a local dependency on the computer which compiles the project to be 
> > able to do that so that is kind of a dead end. Or is ther

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

Sorry, I missed that. I’m not convinced any of these logs need language agnostic tools for access, but if that’s a goal for other folk I don’t feel strongly about it.On 19 Sep 2024, at 21:06, Štefan Miklošovič  wrote:More to it, it is actually not only about FQL. Audit logging is on Chronicle queues too so inspecting that would be platform independent as well. CEP-12 suggests that there might be a persistent store for diagnostic events as well. If somebody wants to inspect what a node was doing after it went offline as for now all these events are in memory only.This would basically enable people to fully inspect what the cluster was doing from FQL to Audit to Diagnostics in a language independent manner. On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič  wrote:I think the biggest selling point for using something like protobuf is what David said - what if he wants to replay it in Go? Basing it on something language neutral enables people to replay it in whatever they want. If we have something totally custom then it is replayable just in Java without bringing tons of dependencies to their projects. That is the message I got from what he wrote. On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:Do we need any of these things either? We have our own serialisation framework and file readers and writers, and at least in the FQL case these are the native serialisation format. At cursory glance it also looks to me like this would be a minimal refactor from the current state.What is the reason we want to add these other dependencies?On 19 Sep 2024, at 20:31, Štefan Miklošovič  wrote:well the Maven plugin declares that it downloads protoc from Maven Central automatically _somehow_ so coding up an Ant task which does something similar shouldn't be too hard. I will investigate this idea. On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
 wrote:
> Unfortunately there is nothing like that for Ant, protoc would need to be a local dependency on the computer which compiles the project to be able to do that so that is kind of a dead end. Or is there any workaround here?

In the old thrift days I believe we generated the code and checked it
in so you didn't need to compile locally.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

More to it, it is actually not only about FQL. Audit logging is on
Chronicle queues too so inspecting that would be platform independent as
well.

CEP-12 suggests that there might be a persistent store for diagnostic
events as well. If somebody wants to inspect what a node was doing after it
went offline as for now all these events are in memory only.

This would basically enable people to fully inspect what the cluster was
doing from FQL to Audit to Diagnostics in a language independent manner.

On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič 
wrote:

> I think the biggest selling point for using something like protobuf is
> what David said - what if he wants to replay it in Go? Basing it on
> something language neutral enables people to replay it in whatever they
> want. If we have something totally custom then it is replayable just in
> Java without bringing tons of dependencies to their projects. That is the
> message I got from what he wrote.
>
> On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:
>
>> Do we need any of these things either? We have our own serialisation
>> framework and file readers and writers, and at least in the FQL case these
>> are the native serialisation format.
>>
>> At cursory glance it also looks to me like this would be a minimal
>> refactor from the current state.
>>
>> What is the reason we want to add these other dependencies?
>>
>>
>> On 19 Sep 2024, at 20:31, Štefan Miklošovič 
>> wrote:
>>
>> 
>> well the Maven plugin declares that it downloads protoc from Maven
>> Central automatically _somehow_ so coding up an Ant task which does
>> something similar shouldn't be too hard. I will investigate this idea.
>>
>> On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams 
>> wrote:
>>
>>> On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
>>>  wrote:
>>> > Unfortunately there is nothing like that for Ant, protoc would need to
>>> be a local dependency on the computer which compiles the project to be able
>>> to do that so that is kind of a dead end. Or is there any workaround here?
>>>
>>> In the old thrift days I believe we generated the code and checked it
>>> in so you didn't need to compile locally.
>>>
>>> Kind Regards,
>>> Brandon
>>>
>>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

I think the biggest selling point for using something like protobuf is what
David said - what if he wants to replay it in Go? Basing it on something
language neutral enables people to replay it in whatever they want. If we
have something totally custom then it is replayable just in Java without
bringing tons of dependencies to their projects. That is the message I got
from what he wrote.

On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:

> Do we need any of these things either? We have our own serialisation
> framework and file readers and writers, and at least in the FQL case these
> are the native serialisation format.
>
> At cursory glance it also looks to me like this would be a minimal
> refactor from the current state.
>
> What is the reason we want to add these other dependencies?
>
>
> On 19 Sep 2024, at 20:31, Štefan Miklošovič 
> wrote:
>
> 
> well the Maven plugin declares that it downloads protoc from Maven Central
> automatically _somehow_ so coding up an Ant task which does something
> similar shouldn't be too hard. I will investigate this idea.
>
> On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:
>
>> On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
>>  wrote:
>> > Unfortunately there is nothing like that for Ant, protoc would need to
>> be a local dependency on the computer which compiles the project to be able
>> to do that so that is kind of a dead end. Or is there any workaround here?
>>
>> In the old thrift days I believe we generated the code and checked it
>> in so you didn't need to compile locally.
>>
>> Kind Regards,
>> Brandon
>>
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

Do we need any of these things either? We have our own serialisation framework and file readers and writers, and at least in the FQL case these are the native serialisation format. At cursory glance it also looks to me like this would be a minimal refactor from the current state.What is the reason we want to add these other dependencies?On 19 Sep 2024, at 20:31, Štefan Miklošovič  wrote:well the Maven plugin declares that it downloads protoc from Maven Central automatically _somehow_ so coding up an Ant task which does something similar shouldn't be too hard. I will investigate this idea. On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
 wrote:
> Unfortunately there is nothing like that for Ant, protoc would need to be a local dependency on the computer which compiles the project to be able to do that so that is kind of a dead end. Or is there any workaround here?

In the old thrift days I believe we generated the code and checked it
in so you didn't need to compile locally.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

well the Maven plugin declares that it downloads protoc from Maven Central
automatically _somehow_ so coding up an Ant task which does something
similar shouldn't be too hard. I will investigate this idea.

On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:

> On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
>  wrote:
> > Unfortunately there is nothing like that for Ant, protoc would need to
> be a local dependency on the computer which compiles the project to be able
> to do that so that is kind of a dead end. Or is there any workaround here?
>
> In the old thrift days I believe we generated the code and checked it
> in so you didn't need to compile locally.
>
> Kind Regards,
> Brandon
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread Brandon Williams

On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
 wrote:
> Unfortunately there is nothing like that for Ant, protoc would need to be a 
> local dependency on the computer which compiles the project to be able to do 
> that so that is kind of a dead end. Or is there any workaround here?

In the old thrift days I believe we generated the code and checked it
in so you didn't need to compile locally.

Kind Regards,
Brandon

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

Great stuff. Keep it going. If we go to replace this then posts like this
are great to gather the feedback.

I like the protobuf idea. If we were on Maven then we could use the
protobuf plugin which compiles the schema as part of the build and it
generates Java classes from it which we might use for actual query
serialization into binary format.

Unfortunately there is nothing like that for Ant, protoc would need to be a
local dependency on the computer which compiles the project to be able to
do that so that is kind of a dead end. Or is there any workaround here?

On Thu, Sep 19, 2024 at 8:14 PM David Capwell  wrote:

> I personally don’t mind switching off Chronicle Queue.  I have a
> transformer function to convert the FQL logs to Thrift (don’t judge) and
> use easy-cas to reply on a cluster… replying FQL from Chronicle Queue was
> far slower than Thrift and was hard to push the cluster as the client was
> the bottleneck… switching off it let me actually cause Cassandra to be the
> bottleneck…
>
> No, there is another perfectly sensible option: just implement a simple
> serialisation format ourselves.
>
>
> My one issue with this is we need to ask who the target audience is?
> Trying to add FQL reply to easy-cas was a pain for 2 reasons: Chronicle
> Queue is slow, custom C* serializers that must be in the class path (that
> brings a ton of baggage with it)…
>
> For me FQL has 2 use cases
>
> 1) analytic, what are people actually doing and what are there frequencies?
> 2) reply
>
> In both cases custom serializers are a pain due to baggage they bring and
> the limiting nature of it… what if I want a Go based FQL reply?  I need
> java code from cassandra-all…
>
> I personally favor serializers like protobuf/thrift as they are portable
> and can be used by users without issues.  As for the log format itself… a
> super simple log format that can be easy to read that is custom is fine by
> me… I am cool with the log being custom as I don’t know a good portable log
> format at the top of my head… a simple thing like the following works for me
>
> Header: lengths, checksum, etc.
> Body: std serializer
> +
>
> On Sep 19, 2024, at 9:14 AM, C. Scott Andreas 
> wrote:
>
> Agree with Benedict's proposal here.
>
> In circumstances when I've needed to capture and work with FQL, I've found
> it cumbersome to work with Chronicle. The dial-home functionality and
> release process changes put it over the top for me.
>
> – Scott
>
> On Sep 19, 2024, at 8:40 AM, Josh McKenzie  wrote:
>
>
> there is another perfectly sensible option
>
> My apologies; I wasn't clear. *If we choose to continue to use chronicle
> queue*, what I enumerated was the only logical option I saw for us.
>
> Altogether I think we should just move away from the library as you've
> laid out here Benedict.
>
> On Thu, Sep 19, 2024, at 11:34 AM, Benedict wrote:
>
>
> No, there is another perfectly sensible option: just implement a simple
> serialisation format ourselves.
>
> I am against forking their code; that is a much higher maintenance burden
> than just writing something simple ourselves. We’ve spent longer
> collectively discussing and maintaining this dependency than it would take
> to implement the features we use.
>
> I still have not heard a compelling reason we adopted it as a dependency
> in the first place.
>
> On 19 Sep 2024, at 16:26, Josh McKenzie  wrote:
>
> 
>
> a jerk move, but they started it with this weird release model
>
> I think that's the only option given their release model and lack of
> backporting bugfixes to the latest ea. Either you run tip of the spear, pay
> them for bugfixes, or run what's effectively an unsupported LTS in the form
> of ea.
>
> So doesn't seem like a jerk move to me as much as it seems like an
> eventuality of their release model.
>
> On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote:
>
> I feel like a group of us discussed this IRL a bit at ApacheCon in Vegas ~
> 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone explaining
> the string operations overhead in the JVM of log concatenation vs slapping
> binary to CQ’s off heap-and-append operation was substantial.
>
> We could hostile fork and bring the bits we use in tree (a jerk move, but
> they started it with this weird release model). I’d rather avoid this, but
> it’s an option seeing as how it’s ASFv2.
>
> On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan 
> wrote:
>
>
> When it comes to alternatives, what about logback + slf4j? It has
> appenders where we want, it is sync / async, we can code some nio appender
> too I guess, it logs it as text into a file so we do not need any special
> tooling to review that. For tailing which Chronicle also offers, I guess
> "tail -f that.log" just does the job? logback even rolls the files after
> they are big enough so it rolls the files the same way after some
> configured period / size as Chronicle does (It even compresses the logs).
>
>
> Yes it was considered.  The whole point was to have a bi

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread David Capwell

I personally don’t mind switching off Chronicle Queue.  I have a transformer 
function to convert the FQL logs to Thrift (don’t judge) and use easy-cas to 
reply on a cluster… replying FQL from Chronicle Queue was far slower than 
Thrift and was hard to push the cluster as the client was the bottleneck… 
switching off it let me actually cause Cassandra to be the bottleneck…

>>> No, there is another perfectly sensible option: just implement a simple 
>>> serialisation format ourselves.

My one issue with this is we need to ask who the target audience is?  Trying to 
add FQL reply to easy-cas was a pain for 2 reasons: Chronicle Queue is slow, 
custom C* serializers that must be in the class path (that brings a ton of 
baggage with it)… 

For me FQL has 2 use cases

1) analytic, what are people actually doing and what are there frequencies?
2) reply

In both cases custom serializers are a pain due to baggage they bring and the 
limiting nature of it… what if I want a Go based FQL reply?  I need java code 
from cassandra-all…

I personally favor serializers like protobuf/thrift as they are portable and 
can be used by users without issues.  As for the log format itself… a super 
simple log format that can be easy to read that is custom is fine by me… I am 
cool with the log being custom as I don’t know a good portable log format at 
the top of my head… a simple thing like the following works for me

Header: lengths, checksum, etc.
Body: std serializer
+

> On Sep 19, 2024, at 9:14 AM, C. Scott Andreas  wrote:
> 
> Agree with Benedict's proposal here.
> 
> In circumstances when I've needed to capture and work with FQL, I've found it 
> cumbersome to work with Chronicle. The dial-home functionality and release 
> process changes put it over the top for me.
> 
> – Scott
> 
>> On Sep 19, 2024, at 8:40 AM, Josh McKenzie  wrote:
>> 
>> 
>>> there is another perfectly sensible option
>> My apologies; I wasn't clear. If we choose to continue to use chronicle 
>> queue, what I enumerated was the only logical option I saw for us.
>> 
>> Altogether I think we should just move away from the library as you've laid 
>> out here Benedict.
>> 
>> On Thu, Sep 19, 2024, at 11:34 AM, Benedict wrote:
>>> 
>>> No, there is another perfectly sensible option: just implement a simple 
>>> serialisation format ourselves.
>>> 
>>> I am against forking their code; that is a much higher maintenance burden 
>>> than just writing something simple ourselves. We’ve spent longer 
>>> collectively discussing and maintaining this dependency than it would take 
>>> to implement the features we use.
>>> 
>>> I still have not heard a compelling reason we adopted it as a dependency in 
>>> the first place.
>>> 
 On 19 Sep 2024, at 16:26, Josh McKenzie  wrote:

> a jerk move, but they started it with this weird release model
 I think that's the only option given their release model and lack of 
 backporting bugfixes to the latest ea. Either you run tip of the spear, 
 pay them for bugfixes, or run what's effectively an unsupported LTS in the 
 form of ea.

 So doesn't seem like a jerk move to me as much as it seems like an 
 eventuality of their release model.

 On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote:
> I feel like a group of us discussed this IRL a bit at ApacheCon in Vegas 
> ~ 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone 
> explaining the string operations overhead in the JVM of log concatenation 
> vs slapping binary to CQ’s off heap-and-append operation was substantial. 
> 
> We could hostile fork and bring the bits we use in tree (a jerk move, but 
> they started it with this weird release model). I’d rather avoid this, 
> but it’s an option seeing as how it’s ASFv2. 
> 
> On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan 
> mailto:[email protected]>> wrote:
> 
>> When it comes to alternatives, what about logback + slf4j? It has 
>> appenders where we want, it is sync / async, we can code some nio 
>> appender too I guess, it logs it as text into a file so we do not need 
>> any special tooling to review that. For tailing which Chronicle also 
>> offers, I guess "tail -f that.log" just does the job? logback even rolls 
>> the files after they are big enough so it rolls the files the same way 
>> after some configured period / size as Chronicle does (It even 
>> compresses the logs).
> 
> 
> Yes it was considered.  The whole point was to have a binary log because 
> serialization to/from (remember replay is part off this) text explodes 
> the size on disk and in memory as well as the processing time required 
> and does not meet the timing requirements of fqltool.
> 
> -Jeremiah

>> 
> 
>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread C. Scott Andreas

Agree with Benedict's proposal here. In circumstances when I've needed to capture and work with FQL, I've
found it cumbersome to work with Chronicle. The dial-home functionality and release process changes put it
over the top for me. – Scott On Sep 19, 2024, at 8:40 AM, Josh McKenzie wrote:
there is another perfectly sensible option My apologies; I wasn't clear. If we choose to continue to use
chronicle queue , what I enumerated was the only logical option I saw for us. Altogether I think we should
just move away from the library as you've laid out here Benedict. On Thu, Sep 19, 2024, at 11:34 AM,
Benedict wrote: No, there is another perfectly sensible option: just implement a simple serialisation
format ourselves. I am against forking their code; that is a much higher maintenance burden than just
writing something simple ourselves. We’ve spent longer collectively discussing and maintaining this
dependency than it would take to implement the features we use. I still have not heard a compelling reason
we adopted it as a dependency in the first place. On 19 Sep 2024, at 16:26, Josh McKenzie
wrote: a jerk move, but they started it with this weird release model I think
that's the only option given their release model and lack of backporting bugfixes to the latest ea. Either
you run tip of the spear, pay them for bugfixes, or run what's effectively an unsupported LTS in the form
of ea. So doesn't seem like a jerk move to me as much as it seems like an eventuality of their release
model. On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote: I feel like a group of us discussed this IRL a
bit at ApacheCon in Vegas ~ 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone explaining the
string operations overhead in the JVM of log concatenation vs slapping binary to CQ’s off heap-and-append
operation was substantial. We could hostile fork and bring the bits we use in tree (a jerk move, but they
started it with this weird release model). I’d rather avoid this, but it’s an option seeing as how it’s
ASFv2. On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan < [email protected] > wrote: When it
comes to alternatives, what about logback + slf4j? It has appenders where we want, it is sync / async, we
can code some nio appender too I guess, it logs it as text into a file so we do not need any special
tooling to review that. For tailing which Chronicle also offers, I guess "tail -f that.log" just
does the job? logback even rolls the files after they are big enough so it rolls the files the same way
after some configured period / size as Chronicle does (It even compresses the logs). Yes it was considered.
The whole point was to have a binary log because serialization to/from (remember replay is part off this)
text explodes the size on disk and in memory as well as the processing time required and does not meet the
timing requirements of fqltool. -Jeremiah

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

2024-09-19 Thread Josh McKenzie

> there is another perfectly sensible option
My apologies; I wasn't clear. *If we choose to continue to use chronicle 
queue*, what I enumerated was the only logical option I saw for us.

Altogether I think we should just move away from the library as you've laid out 
here Benedict.

On Thu, Sep 19, 2024, at 11:34 AM, Benedict wrote:
> 
> No, there is another perfectly sensible option: just implement a simple 
> serialisation format ourselves.
> 
> I am against forking their code; that is a much higher maintenance burden 
> than just writing something simple ourselves. We’ve spent longer collectively 
> discussing and maintaining this dependency than it would take to implement 
> the features we use.
> 
> I still have not heard a compelling reason we adopted it as a dependency in 
> the first place.
> 
>> On 19 Sep 2024, at 16:26, Josh McKenzie  wrote:
>> 
>>> a jerk move, but they started it with this weird release model
>> I think that's the only option given their release model and lack of 
>> backporting bugfixes to the latest ea. Either you run tip of the spear, pay 
>> them for bugfixes, or run what's effectively an unsupported LTS in the form 
>> of ea.
>> 
>> So doesn't seem like a jerk move to me as much as it seems like an 
>> eventuality of their release model.
>> 
>> On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote:
>>> I feel like a group of us discussed this IRL a bit at ApacheCon in Vegas ~ 
>>> 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone explaining 
>>> the string operations overhead in the JVM of log concatenation vs slapping 
>>> binary to CQ’s off heap-and-append operation was substantial. 
>>> 
>>> We could hostile fork and bring the bits we use in tree (a jerk move, but 
>>> they started it with this weird release model). I’d rather avoid this, but 
>>> it’s an option seeing as how it’s ASFv2. 
>>> 
>>> On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan  
>>> wrote:
 
> When it comes to alternatives, what about logback + slf4j? It has 
> appenders where we want, it is sync / async, we can code some nio 
> appender too I guess, it logs it as text into a file so we do not need 
> any special tooling to review that. For tailing which Chronicle also 
> offers, I guess "tail -f that.log" just does the job? logback even rolls 
> the files after they are big enough so it rolls the files the same way 
> after some configured period / size as Chronicle does (It even compresses 
> the logs).
 
 Yes it was considered.  The whole point was to have a binary log because 
 serialization to/from (remember replay is part off this) text explodes the 
 size on disk and in memory as well as the processing time required and 
 does not meet the timing requirements of fqltool.
 
 -Jeremiah
>>

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library