I'd put it another way - the scope is to make it possible to provide a new
implementation of sstable format without the necessity to refactor
Cassandra code. It implies a contract about the responsibilities of sstable
format implementation so that the rest of the code can rely on that, and
only on that, and do not make assumptions beyond that. But it does not
claim that the created interfaces will not change even with a minor version
release. When those interfaces are around for sometime, we can start a
separate discussion about whether we want to put some guarantees on them.
- - -- --- ----- -------- -------------
Jacek Lewandowski
On Wed, Nov 10, 2021 at 9:01 PM David Capwell <dcapw...@apple.com.invalid>
wrote:
> If this gets descoped to test only (can break all interfaces in a minor)
> then my support concerns are no longer valid; I am cool with the CEP scoped
> only to improving testing
>
> > On Nov 10, 2021, at 11:20 AM, Jacek Lewandowski <
> lewandowski.ja...@gmail.com> wrote:
> >
> > For the other ticket (schema update handler interface) I was also
> proposing
> > a kind of @DeveloperApi annotation as seen in other projects but
> similarly
> > to this thread there were different opinions and no conclusion. After
> > reading the comments I must agree that perhaps it is way too early to
> mark
> > this interface as stable. Perhaps it was too far-fetched to say it would
> be
> > for people who wished to replace the SSTable format. My focus is
> > primarily on cleaning up the code (modularization and clean contracts)
> and
> > making it possible to introduce a new format in the future while allowing
> > us to maintain the old format (no "if then else" approach)
> >
> > - - -- --- ----- -------- -------------
> > Jacek Lewandowski
> >
> >
> > On Wed, Nov 10, 2021 at 12:53 AM bened...@apache.org <
> bened...@apache.org>
> > wrote:
> >
> >>> I may be wrong here, but the CEP directly calls out making this api
> >> public for people who wish to replace the SSTable format
> >>
> >> I don’t think this implies API stability. For starters, it doesn’t
> >> stipulate that these implementations will be supported out of tree (the
> >> only one I’m aware of, so far as I understand, is intended to be
> incubated
> >> in tree), nor does an API for external usage have to be stable. It’s
> fine
> >> to create an API and tell users it’s unstable, and that they should
> closely
> >> monitor patch version changes if they use it.
> >>
> >> That said, norms may be changing around what can go into patch releases
> >> anyhow, so this may be a lot of noise about nothing. If all new
> development
> >> goes into trunk, then it’s all moot. But I don’t think we can make hard
> >> assumptions about that today, as historically these sorts of intentions
> >> haven’t lasted.
> >>
> >> I’m fairly against the idea of introducing hard restrictions on this,
> and
> >> potentially ossifying the codebase. I’m not keen to even consider out of
> >> tree consumers of these APIs in any way, for compatibility,
> upgradeability
> >> or anything. There’s a lot that needs to be done over the coming years
> to
> >> improve the internal structure of the project, and unduly entrenching
> the
> >> current state of affairs would be a huge potential harm of these
> efforts to
> >> modularise the codebase.
> >>
> >> From: David Capwell <dcapw...@apple.com.INVALID>
> >> Date: Tuesday, 9 November 2021 at 23:38
> >> To: dev@cassandra.apache.org <dev@cassandra.apache.org>
> >> Subject: Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
> >>> My understanding is that the only interface that is expected to be
> >> stable for external consumers is the secondary index API
> >>
> >> I may be wrong here, but the CEP directly calls out making this api
> public
> >> for people who wish to replace the SSTable format ("Cassandra developers
> >> who want to develop and publish different file format
> implementations."),
> >> so if we need to support 2i API, why would we not support SSTable API as
> >> well?
> >>
> >>> All of the other mentioned APIs are in my opinion for internal usage
> only
> >>
> >> This gets back to my point; it is currently tribal knowledge what needs
> to
> >> work and what doesn’t, and without the broader set of committers knowing
> >> this then the likely hood any new API will break in a minor is high.
> >>
> >>> On Nov 9, 2021, at 12:13 PM, bened...@apache.org wrote:
> >>>
> >>> I agree that we don’t need to block the CEP on this, and that we should
> >> have that discussion. But it’s worth noting that the CEP should not
> >> anticipate or depend on any specific outcome of that discussion.
> >>>
> >>> Since it is somewhat relevant for this discussion, my view is that no
> >> interface should be assumed to be stable without the prior explicit
> >> agreement of the community.
> >>>
> >>> My understanding is that the only interface that is expected to be
> >> stable for external consumers is the secondary index API. Perhaps also
> >> snitches? But also perhaps not, as the difficulty of upgrading these at
> the
> >> same time is pretty low for custom snitches. All of the other mentioned
> >> APIs are in my opinion for internal usage only, so users should not
> assume
> >> compile time compatibility across any release, and I am certain we have
> >> never tried to maintained this. This still facilitates forks of course,
> by
> >> localising the compatibility work.
> >>>
> >>>
> >>> From: Jeremiah D Jordan <jeremiah.jor...@gmail.com>
> >>> Date: Tuesday, 9 November 2021 at 19:43
> >>> To: Cassandra DEV <dev@cassandra.apache.org>
> >>> Subject: Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)
> >>> I would love to have this discussion and setup annotations or similar
> to
> >> formalize things. I just do not think we need to hold any up CEPs to do
> >> so. That discussion should possibly be a CEP of its own proposing how
> we
> >> want to formalize interfaces? I would be happy to go through and try to
> >> put together something for that or since you feel so strongly about it
> >> maybe you want to David? At the very least it should get its own
> DISCUSS
> >> thread and then be written up in the wiki.
> >>>
> >>> -Jeremiah
> >>>
> >>>> On Nov 9, 2021, at 1:06 PM, Joshua McKenzie <jmcken...@apache.org>
> >> wrote:
> >>>>
> >>>>>
> >>>>> trunk -> anything goes, not trunk -> try not to change these
> interfaces
> >>>>
> >>>> Have we ever clarified what "these interfaces" are? Was just talking
> to
> >>>> David and I realized I didn't even JavaDoc CommitLogReadHandler as
> >> _being
> >>>> designed_ for external usage. /sigh
> >>>>
> >>>> I think it'd be valuable for us to go through the codebase and
> annotate
> >>>> interfaces as intended to be exposed to 3rd parties; this has bothered
> >> me
> >>>> for years. Especially as we come up on a large number of new cleanups,
> >>>> refactorings, and potentially genericizing some subsystems into API's
> >>>> (CEP-18 descendents).
> >>>>
> >>>>
> >>>> On Tue, Nov 9, 2021 at 2:01 PM David Capwell
> <dcapw...@apple.com.invalid
> >>>
> >>>> wrote:
> >>>>
> >>>>>> We already have many interfaces similar to these for Compaction
> >>>>> Strategy, Indexing, Query Handler.
> >>>>>
> >>>>> Today-I-Learned QueryHandler is not allowed to be touched in a minor…
> >> good
> >>>>> to know…
> >>>>>
> >>>>>> not trunk -> try not to change these interfaces
> >>>>>
> >>>>> Outside of MBeans, I honestly do not know what interfaces fall into
> >> this
> >>>>> group; and for MBeans we have tests which block breaking changes.
> The
> >>>>> point I am making is that not everyone is aware of the rules, so
> having
> >>>>> something in place to help enforce such rules should be thought
> about;
> >> if
> >>>>> we want to add pluggable hooks with the intent that external parties
> >> can
> >>>>> leverage such hooks, we should also add to the scope the maintenance
> of
> >>>>> these interfaces (we should not assume “tribal knowledge” will work).
> >>>>>
> >>>>> I am not trying to ask for something large or something requiring a
> >> ton of
> >>>>> work, I am just asking that this gets thought about during the
> project
> >> so
> >>>>> it doesn’t get neglected. This could be as simple as an annotation
> >> like
> >>>>> @ExposedTo3rdParties (Hadoop does this to show an interface is
> exposed
> >> and
> >>>>> must be maintained), or it could be something like split directories
> >>>>> (src/java = private, src/java-exposed = public); I am trying not to
> >> dictate
> >>>>> an implementation, only trying to make sure we are setup to support
> >> the CEP
> >>>>> after the work is done.
> >>>>>
> >>>>>
> >>>>>> On Nov 9, 2021, at 9:52 AM, Jeremiah D Jordan <
> >> jeremiah.jor...@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> We already have many interfaces similar to these for Compaction
> >>>>> Strategy, Indexing, Query Handler. I would hope that commiters are
> >> already
> >>>>> following a policy along the lines of trunk -> anything goes, not
> >> trunk ->
> >>>>> try not to change these interfaces. I would expect that to be the
> same
> >>>>> policy for any new internal interfaces that are added. But given we
> >>>>> already have many such interfaces, I see no reason to block adding
> >> more of
> >>>>> them while change policies are discussed.
> >>>>>>
> >>>>>> -Jeremiah
> >>>>>>
> >>>>>>> On Nov 9, 2021, at 10:44 AM, David Capwell
> >> <dcapw...@apple.com.INVALID>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> I still have one outstanding comment, but this is a comment for
> >> several
> >>>>> of the CEPs being worked on
> >>>>>>>
> >>>>>>>> And last comment, which I have also done in the other modularity
> >>>>> thread… backwards compatibility and maintenance. It is not clear
> right
> >> now
> >>>>> what java interfaces may not break and how we can maintain and extend
> >> such
> >>>>> interfaces in the future. If the goal is to allow 3rd parties to
> >> plugin
> >>>>> and offer new SSTable formats, are we as a project ok with having a
> >> minor
> >>>>> release do a binary or source non-compatible change? If not how do
> we
> >>>>> detect this? Until this problem is solved, I do not think we should
> >> add
> >>>>> any such interfaces.
> >>>>>>>
> >>>>>>> I would love some clarity on this. Specifically, if we assume a
> >> patch
> >>>>> author/reviewers are not familiar with the impact of changes these
> >>>>> interfaces, what happens? Do we have tools to block this? Do we
> >> require
> >>>>> 3rd party authors to create massive shims to deal with every patch
> >> level
> >>>>> version out there? I would love more clarity on how we maintain
> these
> >> new
> >>>>> pluggable interfaces.
> >>>>>>>
> >>>>>>>> On Nov 9, 2021, at 4:45 AM, Branimir Lambov <blam...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Does anyone have any further comments or questions on the
> proposal,
> >> or
> >>>>> are
> >>>>>>>> we ready to move forward to a vote?
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Branimir
> >>>>>>>>
> >>>>>>>> On Tue, Nov 2, 2021 at 7:15 PM David Capwell
> >>>>> <dcapw...@apple.com.invalid>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>>> I apologize I did not mention those things explicitly. All the
> >> places
> >>>>>>>>> where
> >>>>>>>>>> sstable files are accessed directly would have to be refactored.
> >>>>>>>>>
> >>>>>>>>> Works for me
> >>>>>>>>>
> >>>>>>>>>> Speaking about the implementation, one idea I was thinking about
> >> was
> >>>>> that
> >>>>>>>>>> the factories for formats are registered using Java's native
> >> service
> >>>>>>>>>> loader.
> >>>>>>>>>
> >>>>>>>>> I am a fan of ServiceLoader as a means of plugging in.
> >>>>>>>>>
> >>>>>>>>>> I hope this explains a bit
> >>>>>>>>>
> >>>>>>>>> Yep; thanks!
> >>>>>>>>>
> >>>>>>>>>> On Nov 2, 2021, at 1:46 AM, Jacek Lewandowski <
> >>>>>>>>> lewandowski.ja...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> David,
> >>>>>>>>>>
> >>>>>>>>>> I apologize I did not mention those things explicitly. All the
> >> places
> >>>>>>>>> where
> >>>>>>>>>> sstable files are accessed directly would have to be refactored.
> >>>>>>>>>>
> >>>>>>>>>> Regarding TableMetrics - currently it includes many metrics,
> some
> >> of
> >>>>> them
> >>>>>>>>>> are unrelated to sstables at all, but there are metrics which
> are
> >>>>>>>>> specific
> >>>>>>>>>> to the current sstable format, like metrics related to index
> >>>>> summaries or
> >>>>>>>>>> bloom filters. The created gauges query certain methods on
> sstable
> >>>>>>>>> reader -
> >>>>>>>>>> I think the only common metrics for sstables we can leave in
> >>>>> TableMetrics
> >>>>>>>>>> are those for which there are query methods in generic sstable
> >>>>> interface.
> >>>>>>>>>> Other metrics, specific to the certain sstable format should be
> >>>>>>>>> registered
> >>>>>>>>>> by the implementation itself.
> >>>>>>>>>>
> >>>>>>>>>> Speaking about the implementation, one idea I was thinking about
> >> was
> >>>>> that
> >>>>>>>>>> the factories for formats are registered using Java's native
> >> service
> >>>>>>>>>> loader. This way we could get the list of all the factories on
> the
> >>>>>>>>>> classpath and call some method, like `registerMetrics` during
> >> system
> >>>>>>>>>> initialization. That could be also implemented in static
> >> initializer
> >>>>> in
> >>>>>>>>> the
> >>>>>>>>>> factory but it would make it less obvious for the implementors
> >> where
> >>>>> such
> >>>>>>>>>> initialization should be done.
> >>>>>>>>>>
> >>>>>>>>>> I hope this explains a bit
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Jacek
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>
> >>>>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>
