If this gets descoped to test only (can break all interfaces in a minor) then
my support concerns are no longer valid; I am cool with the CEP scoped only to
improving testing
> On Nov 10, 2021, at 11:20 AM, Jacek Lewandowski <lewandowski.ja...@gmail.com>
> wrote:
>
> For the other ticket (schema update handler interface) I was also proposing
> a kind of @DeveloperApi annotation as seen in other projects but similarly
> to this thread there were different opinions and no conclusion. After
> reading the comments I must agree that perhaps it is way too early to mark
> this interface as stable. Perhaps it was too far-fetched to say it would be
> for people who wished to replace the SSTable format. My focus is
> primarily on cleaning up the code (modularization and clean contracts) and
> making it possible to introduce a new format in the future while allowing
> us to maintain the old format (no "if then else" approach)
>
> - - -- --- ----- -------- -------------
> Jacek Lewandowski
>
>
> On Wed, Nov 10, 2021 at 12:53 AM bened...@apache.org <bened...@apache.org>
> wrote:
>
>>> I may be wrong here, but the CEP directly calls out making this api
>> public for people who wish to replace the SSTable format
>>
>> I don’t think this implies API stability. For starters, it doesn’t
>> stipulate that these implementations will be supported out of tree (the
>> only one I’m aware of, so far as I understand, is intended to be incubated
>> in tree), nor does an API for external usage have to be stable. It’s fine
>> to create an API and tell users it’s unstable, and that they should closely
>> monitor patch version changes if they use it.
>>
>> That said, norms may be changing around what can go into patch releases
>> anyhow, so this may be a lot of noise about nothing. If all new development
>> goes into trunk, then it’s all moot. But I don’t think we can make hard
>> assumptions about that today, as historically these sorts of intentions
>> haven’t lasted.
>>
>> I’m fairly against the idea of introducing hard restrictions on this, and
>> potentially ossifying the codebase. I’m not keen to even consider out of
>> tree consumers of these APIs in any way, for compatibility, upgradeability
>> or anything. There’s a lot that needs to be done over the coming years to
>> improve the internal structure of the project, and unduly entrenching the
>> current state of affairs would be a huge potential harm of these efforts to
>> modularise the codebase.
>>
>> From: David Capwell <dcapw...@apple.com.INVALID>
>> Date: Tuesday, 9 November 2021 at 23:38
>> To: dev@cassandra.apache.org <dev@cassandra.apache.org>
>> Subject: Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
>>> My understanding is that the only interface that is expected to be
>> stable for external consumers is the secondary index API
>>
>> I may be wrong here, but the CEP directly calls out making this api public
>> for people who wish to replace the SSTable format ("Cassandra developers
>> who want to develop and publish different file format implementations."),
>> so if we need to support 2i API, why would we not support SSTable API as
>> well?
>>
>>> All of the other mentioned APIs are in my opinion for internal usage only
>>
>> This gets back to my point; it is currently tribal knowledge what needs to
>> work and what doesn’t, and without the broader set of committers knowing
>> this then the likely hood any new API will break in a minor is high.
>>
>>> On Nov 9, 2021, at 12:13 PM, bened...@apache.org wrote:
>>>
>>> I agree that we don’t need to block the CEP on this, and that we should
>> have that discussion. But it’s worth noting that the CEP should not
>> anticipate or depend on any specific outcome of that discussion.
>>>
>>> Since it is somewhat relevant for this discussion, my view is that no
>> interface should be assumed to be stable without the prior explicit
>> agreement of the community.
>>>
>>> My understanding is that the only interface that is expected to be
>> stable for external consumers is the secondary index API. Perhaps also
>> snitches? But also perhaps not, as the difficulty of upgrading these at the
>> same time is pretty low for custom snitches. All of the other mentioned
>> APIs are in my opinion for internal usage only, so users should not assume
>> compile time compatibility across any release, and I am certain we have
>> never tried to maintained this. This still facilitates forks of course, by
>> localising the compatibility work.
>>>
>>>
>>> From: Jeremiah D Jordan <jeremiah.jor...@gmail.com>
>>> Date: Tuesday, 9 November 2021 at 19:43
>>> To: Cassandra DEV <dev@cassandra.apache.org>
>>> Subject: Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
>>> I would love to have this discussion and setup annotations or similar to
>> formalize things. I just do not think we need to hold any up CEPs to do
>> so. That discussion should possibly be a CEP of its own proposing how we
>> want to formalize interfaces? I would be happy to go through and try to
>> put together something for that or since you feel so strongly about it
>> maybe you want to David? At the very least it should get its own DISCUSS
>> thread and then be written up in the wiki.
>>>
>>> -Jeremiah
>>>
>>>> On Nov 9, 2021, at 1:06 PM, Joshua McKenzie <jmcken...@apache.org>
>> wrote:
>>>>
>>>>>
>>>>> trunk -> anything goes, not trunk -> try not to change these interfaces
>>>>
>>>> Have we ever clarified what "these interfaces" are? Was just talking to
>>>> David and I realized I didn't even JavaDoc CommitLogReadHandler as
>> _being
>>>> designed_ for external usage. /sigh
>>>>
>>>> I think it'd be valuable for us to go through the codebase and annotate
>>>> interfaces as intended to be exposed to 3rd parties; this has bothered
>> me
>>>> for years. Especially as we come up on a large number of new cleanups,
>>>> refactorings, and potentially genericizing some subsystems into API's
>>>> (CEP-18 descendents).
>>>>
>>>>
>>>> On Tue, Nov 9, 2021 at 2:01 PM David Capwell <dcapw...@apple.com.invalid
>>>
>>>> wrote:
>>>>
>>>>>> We already have many interfaces similar to these for Compaction
>>>>> Strategy, Indexing, Query Handler.
>>>>>
>>>>> Today-I-Learned QueryHandler is not allowed to be touched in a minor…
>> good
>>>>> to know…
>>>>>
>>>>>> not trunk -> try not to change these interfaces
>>>>>
>>>>> Outside of MBeans, I honestly do not know what interfaces fall into
>> this
>>>>> group; and for MBeans we have tests which block breaking changes. The
>>>>> point I am making is that not everyone is aware of the rules, so having
>>>>> something in place to help enforce such rules should be thought about;
>> if
>>>>> we want to add pluggable hooks with the intent that external parties
>> can
>>>>> leverage such hooks, we should also add to the scope the maintenance of
>>>>> these interfaces (we should not assume “tribal knowledge” will work).
>>>>>
>>>>> I am not trying to ask for something large or something requiring a
>> ton of
>>>>> work, I am just asking that this gets thought about during the project
>> so
>>>>> it doesn’t get neglected. This could be as simple as an annotation
>> like
>>>>> @ExposedTo3rdParties (Hadoop does this to show an interface is exposed
>> and
>>>>> must be maintained), or it could be something like split directories
>>>>> (src/java = private, src/java-exposed = public); I am trying not to
>> dictate
>>>>> an implementation, only trying to make sure we are setup to support
>> the CEP
>>>>> after the work is done.
>>>>>
>>>>>
>>>>>> On Nov 9, 2021, at 9:52 AM, Jeremiah D Jordan <
>> jeremiah.jor...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> We already have many interfaces similar to these for Compaction
>>>>> Strategy, Indexing, Query Handler. I would hope that commiters are
>> already
>>>>> following a policy along the lines of trunk -> anything goes, not
>> trunk ->
>>>>> try not to change these interfaces. I would expect that to be the same
>>>>> policy for any new internal interfaces that are added. But given we
>>>>> already have many such interfaces, I see no reason to block adding
>> more of
>>>>> them while change policies are discussed.
>>>>>>
>>>>>> -Jeremiah
>>>>>>
>>>>>>> On Nov 9, 2021, at 10:44 AM, David Capwell
>> <dcapw...@apple.com.INVALID>
>>>>> wrote:
>>>>>>>
>>>>>>> I still have one outstanding comment, but this is a comment for
>> several
>>>>> of the CEPs being worked on
>>>>>>>
>>>>>>>> And last comment, which I have also done in the other modularity
>>>>> thread… backwards compatibility and maintenance. It is not clear right
>> now
>>>>> what java interfaces may not break and how we can maintain and extend
>> such
>>>>> interfaces in the future. If the goal is to allow 3rd parties to
>> plugin
>>>>> and offer new SSTable formats, are we as a project ok with having a
>> minor
>>>>> release do a binary or source non-compatible change? If not how do we
>>>>> detect this? Until this problem is solved, I do not think we should
>> add
>>>>> any such interfaces.
>>>>>>>
>>>>>>> I would love some clarity on this. Specifically, if we assume a
>> patch
>>>>> author/reviewers are not familiar with the impact of changes these
>>>>> interfaces, what happens? Do we have tools to block this? Do we
>> require
>>>>> 3rd party authors to create massive shims to deal with every patch
>> level
>>>>> version out there? I would love more clarity on how we maintain these
>> new
>>>>> pluggable interfaces.
>>>>>>>
>>>>>>>> On Nov 9, 2021, at 4:45 AM, Branimir Lambov <blam...@apache.org>
>>>>> wrote:
>>>>>>>>
>>>>>>>> Does anyone have any further comments or questions on the proposal,
>> or
>>>>> are
>>>>>>>> we ready to move forward to a vote?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Branimir
>>>>>>>>
>>>>>>>> On Tue, Nov 2, 2021 at 7:15 PM David Capwell
>>>>> <dcapw...@apple.com.invalid>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>> I apologize I did not mention those things explicitly. All the
>> places
>>>>>>>>> where
>>>>>>>>>> sstable files are accessed directly would have to be refactored.
>>>>>>>>>
>>>>>>>>> Works for me
>>>>>>>>>
>>>>>>>>>> Speaking about the implementation, one idea I was thinking about
>> was
>>>>> that
>>>>>>>>>> the factories for formats are registered using Java's native
>> service
>>>>>>>>>> loader.
>>>>>>>>>
>>>>>>>>> I am a fan of ServiceLoader as a means of plugging in.
>>>>>>>>>
>>>>>>>>>> I hope this explains a bit
>>>>>>>>>
>>>>>>>>> Yep; thanks!
>>>>>>>>>
>>>>>>>>>> On Nov 2, 2021, at 1:46 AM, Jacek Lewandowski <
>>>>>>>>> lewandowski.ja...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> David,
>>>>>>>>>>
>>>>>>>>>> I apologize I did not mention those things explicitly. All the
>> places
>>>>>>>>> where
>>>>>>>>>> sstable files are accessed directly would have to be refactored.
>>>>>>>>>>
>>>>>>>>>> Regarding TableMetrics - currently it includes many metrics, some
>> of
>>>>> them
>>>>>>>>>> are unrelated to sstables at all, but there are metrics which are
>>>>>>>>> specific
>>>>>>>>>> to the current sstable format, like metrics related to index
>>>>> summaries or
>>>>>>>>>> bloom filters. The created gauges query certain methods on sstable
>>>>>>>>> reader -
>>>>>>>>>> I think the only common metrics for sstables we can leave in
>>>>> TableMetrics
>>>>>>>>>> are those for which there are query methods in generic sstable
>>>>> interface.
>>>>>>>>>> Other metrics, specific to the certain sstable format should be
>>>>>>>>> registered
>>>>>>>>>> by the implementation itself.
>>>>>>>>>>
>>>>>>>>>> Speaking about the implementation, one idea I was thinking about
>> was
>>>>> that
>>>>>>>>>> the factories for formats are registered using Java's native
>> service
>>>>>>>>>> loader. This way we could get the list of all the factories on the
>>>>>>>>>> classpath and call some method, like `registerMetrics` during
>> system
>>>>>>>>>> initialization. That could be also implemented in static
>> initializer
>>>>> in
>>>>>>>>> the
>>>>>>>>>> factory but it would make it less obvious for the implementors
>> where
>>>>> such
>>>>>>>>>> initialization should be done.
>>>>>>>>>>
>>>>>>>>>> I hope this explains a bit
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Jacek
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>
>>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org
