Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

2019-08-15 Thread Hyukjin Kwon
Yeah, we will probably drop Python 2 entirely after 3.0.0. Python 2 is already deprecated. On Thu, 15 Aug 2019, 18:25 Driesprong, Fokko, wrote: > Sorry for the late reply, was a bit busy lately, but I still would like to > share my thoughts on this. > > For Apache Airflow we're dropping support

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

2019-08-15 Thread Hyukjin Kwon
I mean python 2 _will be_ deprecated in Spark 3. On Thu, 15 Aug 2019, 18:37 Hyukjin Kwon, wrote: > Yeah, we will probably drop Python 2 entirely after 3.0.0. Python 2 is > already deprecated. > > On Thu, 15 Aug 2019, 18:25 Driesprong, Fokko, > wrote: > >> Sorry for the late reply, was a bit

Re: Ask for ARM CI for spark

2019-08-15 Thread Sean Owen
I think the right goal is to fix the remaining issues first. If we set up CI/CD it will only tell us there are still some test failures. If it's stable, and not hard to add to the existing CI/CD, yes it could be done automatically later. You can continue to test on ARM independently for now. It

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

2019-08-15 Thread Driesprong, Fokko
Sorry for the late reply, was a bit busy lately, but I still would like to share my thoughts on this. For Apache Airflow we're dropping support for Python 2 in the next major release. We're now supporting Python 3.5+. Mostly because: - Easier to maintain and test, and less if/else

Re: Ask for ARM CI for spark

2019-08-15 Thread Tianhua huang
Hi all, I want to discuss spark ARM CI again, we took some tests on arm instance based on master and the job includes https://github.com/theopenlab/spark/pull/13 and k8s integration https://github.com/theopenlab/spark/pull/17/ , there are several things I want to talk about: First, about the

Re: Release Apache Spark 2.4.4

2019-08-15 Thread Sean Owen
While we're on the topic: In theory, branch 2.3 is meant to be unsupported as of right about now. There are 69 fixes in branch 2.3 since 2.3.3 was released in Februrary: https://issues.apache.org/jira/projects/SPARK/versions/12344844 Some look moderately important. Should we also, or first,

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

2019-08-15 Thread Shane Knapp
just got an update: there was a problem w/the replacement part, and they're trying to fix it. if that's successful, the expect to have power restored within the hour. if that doesn't work, a new (new) replacement part is scheduled to arrive at 8am tomorrow. shane On Thu, Aug 15, 2019 at 2:07

Re: Release Apache Spark 2.4.4

2019-08-15 Thread Dongjoon Hyun
+1 for that. Kazuaki volunteered for 2.3.4 release last month. AFAIK, he has been preparing that. - https://lists.apache.org/thread.html/6fafeefb7715e8764ccfe5d30c90d7444378b5f4f383ec95e2f1d7de@%3Cdev.spark.apache.org%3E I believe we can handle them after 2.4.4 RC1 (or concurrently.) Hi,

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

2019-08-15 Thread Shane Knapp
quick update: it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored. once i hear something i will let everyone know what's up. On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp wrote: > the berkeley colo had a major power

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

2019-08-15 Thread Wenchen Fan
Thanks for tracking it Shane! On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp wrote: > just got an update: > > there was a problem w/the replacement part, and they're trying to fix it. > if that's successful, the expect to have power restored within the hour. > > if that doesn't work, a new (new)

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

2019-08-15 Thread Shane Knapp
a couple of workers needed a bit more time to finish booting up, so no need for my excursion tomorrow. :) builds be building, things look happy. On Thu, Aug 15, 2019 at 6:46 PM Shane Knapp wrote: > it's back up! some of the workers didn't come back cleanly, so i'll have > to hit up the colo

Re: Ask for ARM CI for spark

2019-08-15 Thread Tianhua huang
@Sean Owen , thanks for your reply. I agree with you basically, two points I have to say :) First, maybe I didn't express clear enough, now we download from Maven Central in our test system, seems the community jenkins ci tests never download the jar packages from maven centry repo, our question

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

2019-08-15 Thread Shane Knapp
it's back up! some of the workers didn't come back cleanly, so i'll have to hit up the colo tomorrow and persuade them in person. On Thu, Aug 15, 2019 at 6:45 PM Wenchen Fan wrote: > Thanks for tracking it Shane! > > On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp wrote: > >> just got an update:

Re: Ask for ARM CI for spark

2019-08-15 Thread bo zhaobo
Hi Sean, Thanks very much for pointing out the roadmap. ;-). Then I think we will continue to focus on our test environment. For the networking problems, I mean that we can access Maven Central, and jobs cloud download the required jar package with a high network speed. What we want to know is

Re: Ask for ARM CI for spark

2019-08-15 Thread Sean Owen
I'm not sure what you mean. The dependencies are downloaded by SBT and Maven like in any other project, and nothing about it is specific to Spark. The worker machines cache artifacts that are downloaded from these, but this is a function of Maven and SBT, not Spark. You may find that the initial