I love the idea. I took a quick look at /databricks/spark-pr-dashboard <https://github.com/databricks/spark-pr-dashboard> and that looks like a nice easy start. It would be interesting to try to make a generic project out of it that would work on top of any project/repo. Basically just refactor all of the Spark-specific code and configurations into a `config.py` (assuming there's not too much frontend Spark-specific code...).
Of course this tool assumes Jira+Github which works for Airflow, but probably isn't as common of a setup to really generalize beyond Apache. It seems like by embracing Github issues and dropping Jira we could be building something much more relevant. Max On Fri, Aug 24, 2018 at 7:27 PM Holden Karau <[email protected]> wrote: > Update: we can do this with the dashboard code. Since this would modify > the JIRAs I’d love a sign-off on turning that feature on from someone on > the PMC (or at least a week with no PMC folks saying no). > > On Thu, Aug 23, 2018 at 8:00 AM Holden Karau <[email protected]> wrote: > >> I mean a few ASF projects update JIRA tickets based on PRs automatically. >> Switching from JIRA to GH issues (or back) is super painful, so I’d >> probably do more incremental improvements personally, I just don’t have the >> time to do something like that. >> >> I’ll take a look at some the K8s tools (over in Beam they’re looking at >> one of the review tagging tools out of K8s) if the Spark one is too >> difficult to adapt to our use case. >> >> On Thu, Aug 23, 2018 at 2:34 AM Eamon Keane <[email protected]> >> wrote: >> >>> Kubernetes is a good place to look as they've invested a lot in github >>> bots >>> and label based workflows. E.g. the cherry-picking script and doc is >>> here: >>> >>> >>> https://github.com/kubernetes/kubernetes/blob/master/hack/cherry_pick_pull.sh >>> >>> https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md >>> >>> And more general overview here: >>> >>> https://github.com/kubernetes/community/tree/master/contributors/devel >>> >>> On Thu, Aug 23, 2018 at 5:11 AM Maxime Beauchemin < >>> [email protected]> wrote: >>> >>> > I've heard many times in the past about a GH/Jira syncing tool but >>> never >>> > seen it in action. Personally my vote is to move issues to GH and drop >>> > Jira. Though in the process this will break the release helper script >>> here: >>> > >>> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-jira >>> > >>> > We'll be working on a Github label-driven release baking magic script >>> for >>> > Superset, maybe we could use the same tooling on both Airflow and >>> Superset. >>> > The idea is that the script would use labels like `target:apache-1.11` >>> on >>> > PRs to bake releases. The tool would take as input a base SHA and >>> release >>> > minor release number, and would craft a release branch, fetch and >>> > cherry-pick all the right commits in the right order based on labels, >>> > generate release tags (on minor versions) and output state into >>> > release-info files (listing the base, all cherries, all tags, ...). The >>> > tricky part is resolving merge conflicts while auto-picking cherries, >>> but >>> > the script would guide the operator through it . >>> > >>> > Curious to hear about how other projects do it. I think it generally >>> > involves a lot of manual work. Let me know if you know of open source >>> > tooling to deal with release management. >>> > >>> > Max >>> > >>> > On Wed, Aug 22, 2018 at 6:25 PM Holden Karau <[email protected]> >>> > wrote: >>> > >>> > > Thanks for the reminder, forgot to ask at coffee but I'll ask. >>> > > >>> > > On Wed, Aug 22, 2018, 1:52 AM Driesprong, Fokko <[email protected] >>> > >>> > > wrote: >>> > > >>> > > > Hi Holden, >>> > > > >>> > > > Just curious if you got a hold of someone at the coffee machine :-) >>> > > > >>> > > > Cheers, Fokko >>> > > > >>> > > > Op di 7 aug. 2018 om 09:17 schreef Holden Karau < >>> [email protected] >>> > >: >>> > > > >>> > > > > The JIRA/Github integration tooling I’m a little more fuzzy on >>> but >>> > I’m >>> > > > > doing coffee with some of the folks who probably know the details >>> > this >>> > > > week >>> > > > > and I’ll report back. >>> > > > > >>> > > > > On Tue, Aug 7, 2018 at 12:15 AM Driesprong, Fokko >>> > <[email protected] >>> > > > >>> > > > > wrote: >>> > > > > >>> > > > > > Hi Holden, >>> > > > > > >>> > > > > > Thanks for reaching out. Recently we've moved to Apache Gitbox >>> ( >>> > > > > > https://gitbox.apache.org/), so we use the Github UI directly >>> > > instead >>> > > > of >>> > > > > > having to merge using a CLI ( >>> > > > > > >>> > > >>> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-pr >>> > > > ). >>> > > > > > >>> > > > > > Not sure if we're already up to the game of the dashboard, >>> which >>> > > looks >>> > > > > > awesome btw. But as you also mentioned in your live Airflow PR, >>> > we're >>> > > > > > missing some automation of communication between Jira and >>> Github. >>> > For >>> > > > > > example, as you mentioned, when a PR is opened, the status >>> > > > automagically >>> > > > > > changes to In Progress. Do you have any pointer of how this is >>> set >>> > up >>> > > > at, >>> > > > > > for example the Spark or Beam project? So we can replicate >>> this in >>> > > > > Airflow. >>> > > > > > >>> > > > > > Cheers, Fokko >>> > > > > > >>> > > > > > 2018-08-07 7:28 GMT+02:00 Holden Karau <[email protected] >>> >: >>> > > > > > >>> > > > > > > Hi Y'all, >>> > > > > > > >>> > > > > > > One of the comments from my livestream was asking if the >>> code for >>> > > the >>> > > > > > Spark >>> > > > > > > PR review dashboard <http://spark-prs.appspot.com/> is OSS >>> (it >>> > is >>> > > > > > > <https://github.com/databricks/spark-pr-dashboard>), and I >>> have >>> > a >>> > > > fork >>> > > > > > up >>> > > > > > > for Beam, and I was wondering if folks in Airflow would find >>> > > > something >>> > > > > > like >>> > > > > > > this useful? If so I'd be happy to set that up (if not no >>> > stress). >>> > > > > > > >>> > > > > > > Cheers, >>> > > > > > > >>> > > > > > > Holden :) >>> > > > > > > >>> > > > > > > -- >>> > > > > > > Cell : 425-233-8271 >>> > > > > > > >>> > > > > > >>> > > > > -- >>> > > > > Twitter: https://twitter.com/holdenkarau >>> > > > > >>> > > > >>> > > >>> > >>> >> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
