This sounds like we start with opening Github issues for reports only. That works for me as it's in line with my proposal for phased changes. You can wait for a bit to hear about the opinions and open a VOTE thread for procedural change if there is consensus.
On Thu, Feb 5, 2026 at 4:55 AM Tian Gao via dev <[email protected]> wrote: > Hi all, > > Thank you for all the discussions on this matter. I think we can all agree > that people are interested in this matter, no matter what their positions > are. > > Like I said above, what I want is to make progress for a more involved > community, not figuring out the "best solution" and shooting for the moon. > So I have a new proposal that might make more people happy. > > 1. Open github issues. > 2. Make an issue template where similar fields are required as current > JIRA tickets (issue type, component?, affected versions). I added a > question mark here because a reporter might not know how we categorize > "components". I don't know how this is used in our process. If it only > helps triaging, we can add the label later (by committers). > 3. Create labels to categorize issues. For example, "component-xx" for > different components. We can also have labels for different spark versions. > 4. Still require a JIRA ticket for all PRs (for now, let's discuss if we > can relax the restriction in a separate thread). > 5. Build a github bot that can pick up a special label in github issues > ("create-jira" for example), and duplicate that github issue in the JIRA > system. So for all github issues, if we believe it's valid and should be > worked on, all we need to do is to add a label to it, and the same JIRA > issue will be created for all our existing infra. > > I know people have different opinions about how to avoid duplicating > information between JIRA/PR and how to improve merging experience. They are > all kind of relevant to how we use github as a platform, but let's find our > common ground and do it first. We'll have more stat once we move one step > forward. > > Also if we decide to do this, I don't believe it requires an SPIP. It's a > procedural change (and a rather easy one). What should be the process to > land this? > > Tian Gao > > On Sun, Feb 1, 2026 at 5:41 PM vaquar khan <[email protected]> wrote: > >> Hi all, >> >> I've been auditing our dev automation to see how the SPIP impacts the >> release pipeline. We have some heavy JIRA coupling in >> *dev/merge_spark_pr.py *that will break our workflow if we don't handle >> the transition surgically. >> >> Right now, the script is locked into the jira python lib and explicitly >> calls asf_jira.project_versions("SPARK") to fetch unreleased versions. It >> uses specific logic to map branch names like branch-3.5 to JIRA versions >> and filters invalid combinations to keep history clean. This metadata is >> the source of truth for our release notes. If we cut over to GitHub >> milestones without a dual-write setup, we risk "metadata drift" where fixes >> merged to master won't show up in the 3.5.x JIRA reports, making our >> enterprise documentation incomplete. >> >> To address the concerns raised about fragmented releases, I’m proposing a >> "dual-rail" period where we refactor the merge script to update both >> trackers simultaneously. This protects the integrity of the 3.5/4.0 >> maintenance split while we validate the new workflow. >> >> We also need to avoid the data corruption Maven hit during their >> migration. Their script hit a race condition with GitHub's async import API >> returning 404s for "pending" issues, which resulted in thousands of >> duplicate tickets and the accidental deletion of valid PRs during cleanup. >> I’m prototyping a state-aware import tool with a local SQLite cache to >> handle the idempotency so we don't corrupt our historical metadata. >> >> The upside is worth the effort—the Apache Arrow project reported a 39% >> jump in mailing list traffic and hit 2,297 commits in a single quarter >> (approaching their historical highs) after they moved to GitHub Issues. You >> can see the metrics in the official board minutes here: >> https://whimsy.apache.org/board/minutes/Arrow.html. >> >> If the community is interested, I'm happy to help build out this >> migration infra and the hybrid release note generator. I'll wait for >> feedback before I share any design docs on the GraphQL implementation. >> >> Regards, >> Viquar Khan >> https://www.linkedin.com/in/vaquar-khan-b695577/ >> >> On Sun, 1 Feb 2026 at 18:45, Jungtaek Lim <[email protected]> >> wrote: >> >>> Maybe then we could also revisit the purpose of SPIP. >>> >>> https://spark.apache.org/improvement-proposals.html >>> >>> The purpose of an SPIP is to inform and involve the user community in >>>> major improvements to the Spark codebase throughout the development >>>> process, to increase the likelihood that user needs are met. >>>> >>>> SPIPs should be used for significant user-facing or cross-cutting >>>> changes, not small incremental improvements. When in doubt, if a committer >>>> thinks a change needs an SPIP, it does. >>>> >>> IMHO this proposal does not warrant SPIP according to the above as the >>> proposal is nothing to do with Spark "codebase". The first sentence >>> clarifies the purpose of SPIP to be scoped to code changes. It's just that >>> we somehow have been using SPIP for a broader purpose, which I doubt the >>> stricter voting rule considered the case beyond the purpose of SPIP. For me >>> it sounds like an unintentional side effect. Once this proposal no longer >>> couples with SPIP, this will go back to the procedural voting process and >>> veto does not apply. >>> >>> On Mon, Feb 2, 2026 at 6:03 AM Mark Hamstra <[email protected]> >>> wrote: >>> >>>> Ok, perhaps that page should read "...follows the Apache code-change >>>> vote process..." or "...follows the typical Apache code-change >>>> process...", because the clear intent is that the vote is more >>>> restrictive than a procedural vote and that the SPIP cannot pass with >>>> any -1 votes from PMC members: "(at least 3 +1 votes from PMC members >>>> and no -1 votes from PMC members)". >>>> >>>> On Sun, Feb 1, 2026 at 12:47 PM Jungtaek Lim >>>> <[email protected]> wrote: >>>> > >>>> > It says “typical” but the voting rule which enables veto is only for >>>> “code change”. “Typically”, non code change does not come “veto” into play. >>>> > >>>> > https://www.apache.org/foundation/voting.html >>>> > >>>> > So either SPIP overwrites the voting rule to stricter one, or SPIP >>>> does not aim to handle proposal for non code change. Specifically, this >>>> discussion is “procedural” one IMO. >>>> > >>>> > Does it make sense? Which one we were intending to? >>>> > >>>> > 2026년 2월 2일 (월) 오전 12:44, Mark Hamstra <[email protected]>님이 작성: >>>> >> >>>> >> The voting requirements are enumerated in >>>> >> https://spark.apache.org/improvement-proposals.html >>>> >> Is there something unclear there? >>>> >> >>>> >> On Fri, Jan 30, 2026 at 2:51 PM Jungtaek Lim >>>> >> <[email protected]> wrote: >>>> >> > >>>> >> > I'm not sure the voting process for SPIP is intentionally enabling >>>> the veto. It follows the voting process of "code change", but we have non >>>> code change being scoped as SPIP. >>>> >> > >>>> >> > To PMC members - is it intentional for any SPIP including non-code >>>> change to follow the voting process of "code change", or did we imply SPIP >>>> only applies to eventual code change and this topic doesn't warrant SPIP? >>>> >> > >>>> >> > On Sat, Jan 31, 2026 at 7:03 AM Tian Gao <[email protected]> >>>> wrote: >>>> >> >> >>>> >> >> A difference between what Dongjoon proposed and my proposal is - >>>> during this "test phase", is it allowed to submit PRs that are linked to >>>> github issues, instead of JIRA? If it's a yes, then I'm totally fine if we >>>> want to extend this to 3-6 months. If it's a no, I still believe it's a >>>> significant improvement, but we may miss some data about if people feel >>>> more comfortable using github vs JIRA. >>>> >> >> >>>> >> >> If we only allow github issues to be a discussion forum, then I >>>> don't think it deserves an SPIP - let's just open it. >>>> >> >> >>>> >> >> If we want to have both work at the same time (at least start >>>> building infra around github issues), we need to sort out some details - >>>> majorly the procedural differences. What information in JIRA tickets do we >>>> really need so we have to require an equivalent component in github issues. >>>> Can we still do release notes properly. Maybe enforce (highly encourage) >>>> committers to use JIRA during that phase so we still have all the major >>>> pieces the same? >>>> >> >> >>>> >> >> I believe PMC members have the right to veto any SPIP, so I want >>>> to find a common ground here to make some progress. >>>> >> >> >>>> >> >> Tian Gao >>>> >> >> >>>> >> >> On Fri, Jan 30, 2026 at 1:35 PM Jungtaek Lim < >>>> [email protected]> wrote: >>>> >> >>> >>>> >> >>> > Speaking as an occasional contributor, I would expect this to >>>> be more effort than it’s worth. A phased migration is appealing because it >>>> feels safer and more gradual, but I think everyone will be better off in >>>> the long run with a speedy and clear cutover from Jira to GitHub. >>>> >> >>> >>>> >> >>> My main proposal is to construct a way to prove the proposal by >>>> ourselves instead of just arguing around which one is better from >>>> experience with other projects', individual's preference of UI/UX, etc. >>>> Everyone talks from their experience and no one can be on behalf of >>>> artificial potential contributors and other existing contributors >>>> (including committers and PMC members). I'm not sure we don't have good >>>> evidence about changing it as a whole - if all of us had the same >>>> preference, this discussion thread should have just simply been filled with >>>> a wave of +1. That didn't happen. The first phase would give data, at least >>>> for how many issues will be filed from non-code-contributors, which we >>>> collect the accounts and consider these accounts to have been something we >>>> should have handled ASF account creation, or even aggressively, consider >>>> these issues to be non-existed if we didn't migrate. >>>> >> >>> >>>> >> >>> Also if you look at SPIP doc, the plan is already phased. It's >>>> just that 2 weeks is incredibly short for many PMC members, which has a >>>> high chance for them to work nothing about Apache Spark during the time, >>>> and they have a binding vote to make a decision. IMHO it should be much >>>> longer than that, a quarter or a half year. >>>> >> >>> >>>> >> >>> On Sat, Jan 31, 2026 at 3:27 AM Nicholas Chammas < >>>> [email protected]> wrote: >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> > On Jan 30, 2026, at 9:27 AM, Szehon Ho < >>>> [email protected]> wrote: >>>> >> >>>> > >>>> >> >>>> > In my experience, because I see few committers discussing >>>> anything technical on Spark JIRA for years as you mentioned (and other >>>> Hadoop project JIRAs too), I feel like nobody will reply if I do, so I will >>>> make a Github PR directly and ping for feedback there. So in addition to >>>> the UX problem Tian mentioned, it's worsened by cause and effect. So it's >>>> become a procedure, and we still don't have a good place to discuss without >>>> jumping to code. >>>> >> >>>> >>>> >> >>>> This has often been my experience as well. The eyes are mainly >>>> on GitHub and not Jira. >>>> >> >>>> >>>> >> >>>> > On Jan 30, 2026, at 12:12 AM, Jungtaek Lim < >>>> [email protected]> wrote: >>>> >> >>>> > >>>> >> >>>> > Why can't we do this in two phases instead of trying to build >>>> Rome in a day? >>>> >> >>>> >>>> >> >>>> Speaking as an occasional contributor, I would expect this to >>>> be more effort than it’s worth. A phased migration is appealing because it >>>> feels safer and more gradual, but I think everyone will be better off in >>>> the long run with a speedy and clear cutover from Jira to GitHub. The >>>> longer the transitional phase lasts, the more confusing it will be to new >>>> and occasional contributors who are not following the dev process’s >>>> evolution closely. >>>> >> >>>> >>>> >> >>>> Nick >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> --------------------------------------------------------------------- >>>> >> >>>> To unsubscribe e-mail: [email protected] >>>> >> >>>> >>>> >> >>>> >> --------------------------------------------------------------------- >>>> >> To unsubscribe e-mail: [email protected] >>>> >> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: [email protected] >>>> >>>> >> >> -- >> Regards, >> Vaquar Khan >> >>
