Holden, I agree with you a lot in a sense that this is a chicken and egg
situation.
Spark v4.0 release is a really big one, isn't it?
(1) First, do you think the proposed items are 'BLOCKER'-level Apache Spark 4.0
JIRA items? May I ask why you think in that way? If I understand more, I can
help them. For now, as I wrote in "Apache Spark 4.0 Timeframe," there are some
main issues where multiple PMC members called out the necessity of Spark 4.0.0
release as the only feasible path forward. In that level, although I support
the proposed items but those look like nice-to-have items to me because we can
try them in Apache Spark 3.5.0 too if they are ready and mature. To be clear, I
want to say I'm open for the PRs and interested in what we are going to have
with them, Holden.
https://lists.apache.org/thread/3x6dh17bmy20n3frtt3crgxjydnxh2o0
(Apache Spark 3.5.0 Expectations?)
https://lists.apache.org/thread/xhkgj60j361gdpywoxxz7qspp2w80ry6
(Apache Spark 4.0 Timeframe?)
(2) Second, back to the vote, the reason why I proposed the vote on '4.0.0
Plan' is that planning itself has been not considered by default in the
community.
https://lists.apache.org/thread/k7gr65wt0fwtldc7hp7bd0vkg1k93rrb
(ASF policy violation and Scala version issues)
> I will start a vote for Apache Spark 4.0.0 timeframe next week after
> receiving more feedback.
> Since 4.0.0 is not limited to the Scala issues, we will vote on the
> timeline only.
I'm one of the people who feel a responsibility to provide a way to escape this
deadlock situation in the community. It's too easy to say 'No in 3.x era' or to
say 'No 4.0 until my patch is in the promised land'. Without a consensus we
have a 4.0 release, we are only daydreaming about a non-existent Spark 4.0
without any efforts and without knowing why we are blocked. I believe 'the vote
on plan' unleash the community release train in order to re-ignite all
discussion about the previously prohibited items.
https://lists.apache.org/thread/r0zn6rd8y25yn2dg59ktw3ttrwxzqrfb
([VOTE] Release Plan for Apache Spark 4.0.0 (June 2024))
Holden, could you think it in this way too?
Thanks,
Dongjoon.
On 2023/06/12 18:57:32 Holden Karau wrote:
> Yup I think buidling consensus on what goes in 4.X is something we’ll need
> to do.
>
> On Mon, Jun 12, 2023 at 11:56 AM Dongjoon Hyun <dongjoon.h...@gmail.com>
> wrote:
>
> > Thank you for sharing those. I'm also interested in taking advantage of
> > it. Also, I hope `spark-upgrade` can help us in line with Spark 4.0.
> >
> > However, we don't need to discuss any of this if we don't build a
> > consensus on both Spark 4.0 or next Scala version.
> >
> > We don't have a vehicle at all to reach there yet.
> >
> > In the community, I saw a bottleneck; "No in 3.x era" and "No for 4.0 yet
> > because XXX"
> >
> > Dongjoon.
> >
> > --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
