PR builder not working now

2022-04-11 Thread Hyukjin Kwon
Hi all, There is a bug in GitHub Actions' RESTful API (see https://github.com/HyukjinKwon/spark/actions?query=branch%3Adebug-ga-detection as an example). So, currently OSS PR builder doesn't work properly with showing a screen such as

Re: [Spark Core]: Support for un-pivoting data ('melt')

2022-04-11 Thread Enrico Minack
The melt function has recently been implemented in the PySpark Pandas API (because melt is part of the Pandas API). I think, Scala/Java Dataset and Python DataFrame APIs deserve this method equally well, ideally all based on one implementation. I'd like to fuel the conversation with some

Re: Is spark fair scheduler is for kubernete?

2022-04-11 Thread Martin Grigorov
Hi, On Mon, Apr 11, 2022 at 7:43 AM Jason Jun wrote: > the official doc, https://spark.apache.org/docs/latest/job-scheduling.html, > didn't mention that its working for kubernete cluster? > You could use Volcano scheduler for more advanced setups on Kubernetes. Here is an article explaining