ATSv2 Integration

2020-04-21 Thread Daniel Templeton
Hi!  I see that SPARK-1537 is now closed as won't fix.  What was the reason the Spark community decided to skip ATSv2 support? My org is currently looking at how we introduce Spark into YARN clusters that are primarily used for Scalding today.  For the Scalding UI we use an internal tool

Re: Getting the ball started on a 2.4.6 release

2020-04-21 Thread Holden Karau
This seem like not very impactful for end-users on K8s assuming they've got logging of some kind set up. Unless I'm missing something. On Tue, Apr 21, 2020 at 4:51 PM edeesis wrote: > I'd like to advocate for: > > https://issues.apache.org/jira/browse/SPARK-25515 > and >

Re: Getting the ball started on a 2.4.6 release

2020-04-21 Thread edeesis
I'd like to advocate for: https://issues.apache.org/jira/browse/SPARK-25515 and https://issues.apache.org/jira/browse/SPARK-29865 Two small QOL changes that make production use of Spark with Kubernetes much easier. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

2020-04-21 Thread wuyi
I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could cause application hang. And, probably, also https://issues.apache.org/jira/browse/SPARK-31509, to make better guidance of barrier execution for user. But we do not have conclusion yet. Best, Yi Wu -- Sent from: