I think we certainly can't remove it without deprecation and a few releases. If there were big problems with it that weren't getting fixed, sure maybe, but lack of interest in reviewing minor changes isn't necessarily a bad sign. By the same logic you'd delete graphx long ago.
Anecdotally, yes there are people using it that I know of at least, but I wouldn't know a lot of them. I think the question is, is it causing a problem, like a lot of maintenance? doesn't sound like it. On Tue, Sep 15, 2020 at 8:19 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > > Probably it would depend on the meaning of "experimental". My understanding > of "experimental" is more likely "incubation", which may be graduated > finally, or may be retired. > > To be clear, I'm evaluating the continuous mode as "candidate to retire", > unless there are actual use cases in production and at least a couple of > community members volunteer to maintain it. As far as I see the activity in a > year, there's no interest for the continuous mode in community members. I can > refer to at least three PRs which suffered to find reviewers (around 1 year) > and closed on inactivity. No improvements/bug fixes except trivials. It > doesn't seem to get some traction - few questions in SO, a few posts in > google search results which were all posted around the date when continuous > mode was introduced. Though I would be convinced if someone could provide > meaningful numbers of actual use cases. > > If the answer really has to be taken between un-experimental or not (which > says retirement is not an option), I'd rather vote to leave as experimental, > so I just keep forgetting about it. Actually it bothers sometimes even if the > change is done in micro-batch side (so that's not a zero cost to maintain), > but still better than officially supporting it. > > > On Tue, Sep 15, 2020 at 9:08 PM Sean Owen <sro...@gmail.com> wrote: >> >> If you're suggesting making it un-Experimental, probably yes, as it is >> de facto not going to change much I expect. >> If you're saying remove it, probably not? I don't see that it's >> anywhere near deprecated, and not sure it's unmaintained - obviously >> tests etc still have to keep passing. >> >> On Mon, Sep 14, 2020 at 11:34 PM Jungtaek Lim >> <kabhwan.opensou...@gmail.com> wrote: >> > >> > Hi devs, >> > >> > It was Spark 2.3 in Feb 2018 which introduced continuous mode in >> > Structured Streaming as "experimental". >> > >> > Now we are here at 2.5 years after its release - I feel it would be a good >> > time to evaluate the mode, whether the mode has been widely used or not, >> > and the mode has been making progress, as the mode is "experimental". >> > >> > At least from the surface I don't see any active effort for continuous >> > mode around the community - the last major effort was stateful operation >> > which was incomplete and I removed that. There were some couples of bug >> > reports as well as fixes more than a year ago and almost nothing has been >> > handled. (A trivial bugfix PR has been merged recently but that's all.) >> > The new features introduced to the Structured Streaming (at least >> > observable metrics, SS UI) don't apply to continuous mode, and no one made >> > "support continuous mode" as a hard requirement on passing review in these >> > PRs. >> > >> > I have no idea how many companies are using the mode in production (please >> > add the voice if someone has statistics about this) but I don't see any >> > bug reports recently, and see only a few questions in SO, which makes me >> > think about cost on maintenance. >> > >> > I know there's a mood to avoid discontinue support as possible, but it >> > sounds weird to keep something as "unmaintained", especially it's still >> > "experimental" and main authors are no more active enough to promise >> > maintenance/improvement on the module. Thoughts? >> > >> > Thanks, >> > Jungtaek Lim (HeartSaVioR) --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
