Re: Getting the ball started on a 2.4.6 release

2020-04-24 Thread Holden Karau
On Fri, Apr 24, 2020 at 6:14 PM edeesis wrote: > Yes, watching the pod yaml could work for this. Just gotta set up some kind > of thing to do that, thanks for clueing me into that. Sure thing, Kris Nova was the one who clued me into it so just passing it along :) > > > And sounds great re:

Re: Getting the ball started on a 2.4.6 release

2020-04-24 Thread edeesis
Yes, watching the pod yaml could work for this. Just gotta set up some kind of thing to do that, thanks for clueing me into that. And sounds great re: Spark 2.5. Having a transitional release makes sense I think. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread Holden Karau
Sounds good, I’ll make the JIRAs for tracking then and I can ping the original PR authors in their and based on their feedback either include or not. On Thu, Apr 23, 2020 at 11:51 AM Xiao Li wrote: > Actually, SPARK-26390 https://github.com/apache/spark/pull/23343 is just > a small clean up. I

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread Xiao Li
Actually, SPARK-26390 https://github.com/apache/spark/pull/23343 is just a small clean up. I do not think it fixes any correctness bugs. I think we should discuss your backport plans one by one with the PR authors and reviewers, since most of them are not closely following the dev list. Xiao

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread Holden Karau
I included 26390 as a candidate since it sounded like it bordered on a correctness/expected behaviour fix (eg columpruning rule doing more than column pruning), but if it’s too big a change happy to drop that one. On Thu, Apr 23, 2020 at 11:43 AM Xiao Li wrote: > Hi, Holden, > > We are trying

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread Xiao Li
Hi, Holden, We are trying to avoid backporting the improvement/cleanup PRs to the maintenance releases, especially the core modules, like Spark Core and SQL. For example, SPARK-26390 is a good example. Xiao On Thu, Apr 23, 2020 at 11:17 AM Holden Karau wrote: > Tentatively I'm planning on

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread Holden Karau
Tentatively I'm planning on this list to start backporting. If no one sees any issues with those I'll start to make backport JIRAs for them for tracking this afternoon. SPARK-26390 ColumnPruning rule should only do column pruning SPARK-25407 Allow nested access for non-existent field

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread Holden Karau
On Thu, Apr 23, 2020 at 9:07 AM edeesis wrote: > There's other information you can obtain from the Pod metadata on a > describe > than just from the logs, which are typically what's being printed by the > Application itself. Would get pods -w -o yaml do the trick here or is there going to be

Re: Getting the ball started on a 2.4.6 release

2020-04-23 Thread edeesis
There's other information you can obtain from the Pod metadata on a describe than just from the logs, which are typically what's being printed by the Application itself. I've also found that Spark has some trouble obtaining the reason for a K8S executor death (as evident by the

Re: Getting the ball started on a 2.4.6 release

2020-04-22 Thread wuyi
We have a conclusion now and we decide to include SPARK-31509 in the PR of SPARK-31485. So there actually should be only one candidate(But to be honest, it still depends on committers). Best, Yi Wu -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

2020-04-22 Thread Holden Karau
Thanks, I agree improving that error message instead of hanging could be a good candidate for backporting to 2.4 On Tue, Apr 21, 2020 at 6:43 PM wuyi wrote: > I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could > cause application hang. > > > And, probably, also

Re: Getting the ball started on a 2.4.6 release

2020-04-21 Thread wuyi
I have one: https://issues.apache.org/jira/browse/SPARK-31485, which could cause application hang. And, probably, also https://issues.apache.org/jira/browse/SPARK-31509, to make better guidance of barrier execution for user. But we do not have conclusion yet. Best, Yi Wu -- Sent from:

Re: Getting the ball started on a 2.4.6 release

2020-04-21 Thread Holden Karau
This seem like not very impactful for end-users on K8s assuming they've got logging of some kind set up. Unless I'm missing something. On Tue, Apr 21, 2020 at 4:51 PM edeesis wrote: > I'd like to advocate for: > > https://issues.apache.org/jira/browse/SPARK-25515 > and >

Re: Getting the ball started on a 2.4.6 release

2020-04-21 Thread edeesis
I'd like to advocate for: https://issues.apache.org/jira/browse/SPARK-25515 and https://issues.apache.org/jira/browse/SPARK-29865 Two small QOL changes that make production use of Spark with Kubernetes much easier. -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

Re: Getting the ball started on a 2.4.6 release

2020-04-20 Thread Xiao Li
Yes. This one got merged yesterday. Thanks! Xiao On Mon, Apr 20, 2020 at 10:51 AM Sean Owen wrote: > Looks like we have 1 marked for 2.4.6: > https://issues.apache.org/jira/projects/SPARK/versions/12346781 > > https://issues.apache.org/jira/browse/SPARK-31234 ResetCommand should > not wipe

Re: Getting the ball started on a 2.4.6 release

2020-04-20 Thread Sean Owen
Looks like we have 1 marked for 2.4.6: https://issues.apache.org/jira/projects/SPARK/versions/12346781 https://issues.apache.org/jira/browse/SPARK-31234 ResetCommand should not wipe out all configs Xiao might be able to comment on that one. On Mon, Apr 20, 2020 at 11:31 AM Holden Karau wrote: