Re: moving the spark jenkins job builder repo from dbricks --> spark

2018-10-17 Thread shane knapp
t this. shane > On Wed, Oct 10, 2018 at 12:06 PM shane knapp wrote: > >> Not sure if that's what you meant; but it should be ok for the jenkins >>> servers to manually sync with master after you (or someone else) have >>> verified the changes. That should prevent

Re: GitHub is out of order

2018-10-22 Thread shane knapp
월) 오후 12:40, Dongjoon Hyun 님이 작성: >> >>> Hi, All. >>> >>> Currently, GitHub is out of order. Apache Spark repo is also affected. >>> Newly filed pull requests to Apache Spark repository seem to disappear >>> repeatedly, too. >>> >>> https:

Re: GitHub is out of order

2018-10-22 Thread shane knapp
i've actually taken jenkins down completely. after i see an 'all clear' on their status page, i'll bring jenkins back up. On Mon, Oct 22, 2018 at 9:34 AM shane knapp wrote: > jenkins is no longer accepting new builds. > > On Mon, Oct 22, 2018 at 8:55 AM Hyukjin Kwon wrote: > >

Re: GitHub is out of order

2018-10-22 Thread shane knapp
art it back up and keep an eye on PRBs/github statii/etc. shane On Mon, Oct 22, 2018 at 9:57 AM shane knapp wrote: > i've actually taken jenkins down completely. > > after i see an 'all clear' on their status page, i'll bring jenkins back > up. > > On Mon, Oct 22, 2018 at 9:34

Re: GitHub is out of order

2018-10-22 Thread shane knapp
shane knapp wrote: > quick update: > > the github status messages have gone from "red" to "orange", which to me > still means "we're broken". :) > > https://status.github.com/messages > > i'm still holding off on restarting jenkins, but if thin

Re: python lint is broken on master branch

2018-10-31 Thread shane knapp
e > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-lint/9080/console > > > > Any ideas? > > ----- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org &

Re: python lint is broken on master branch

2018-10-31 Thread shane knapp
'. >> > flake8 checks failed. >> > >> > As an example please see >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-lint/9080/console >> > >> > Any ideas? >> >> ---

Re: Java 11 support

2018-11-06 Thread shane knapp
gt; >> > > Flavor: r-devel-linux-x86_64-debian-gcc, r-devel-windows-ix86+x86_64 > >> > > Check: CRAN incoming feasibility, Result: NOTE > >> > > Maintainer: 'Shivaram Venkataraman ' > >> > > > >> > > New submission > >> > > > >> > > Package was archived on CRAN > >> > > > >> > > Possibly mis-spelled words in DESCRIPTION: > >> > > Frontend (4:10, 5:28) > >> > > > >> > > CRAN repository db overrides: > >> > > X-CRAN-Comment: Archived on 2018-05-01 as check problems were not > >> > > corrected despite reminders. > >> > > > >> > > Flavor: r-devel-linux-x86_64-debian-gcc > >> > > Check: re-building of vignette outputs, Result: WARNING > >> > > Error in re-building vignettes: > >> > > ... > >> > > > >> > > Attaching package: 'SparkR' > >> > > > >> > > The following objects are masked from 'package:stats': > >> > > > >> > > cov, filter, lag, na.omit, predict, sd, var, window > >> > > > >> > > The following objects are masked from 'package:base': > >> > > > >> > > as.data.frame, colnames, colnames<-, drop, endsWith, > >> > > intersect, rank, rbind, sample, startsWith, subset, summary, > >> > > transform, union > >> > > > >> > > trying URL ' > http://mirror.klaus-uwe.me/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz > ' > >> > > Content type 'application/octet-stream' length 227893062 bytes > (217.3 MB) > >> > > == > >> > > downloaded 217.3 MB > >> > > > >> > > Quitting from lines 65-67 (sparkr-vignettes.Rmd) > >> > > Error: processing vignette 'sparkr-vignettes.Rmd' failed with > diagnostics: > >> > > Java version 8 is required for this package; found version: 11.0.1 > >> > > Execution halted > >> > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: need assistance debugging a strange build failure...

2018-11-06 Thread shane knapp
btw, this is a compilation error in the SBT build that only shows up on the ubuntu workers. On Mon, Nov 5, 2018 at 5:07 PM shane knapp wrote: > the maven build is quite happy: > > https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7-ubuntu-testing/ > >

need assistance debugging a strange build failure...

2018-11-05 Thread shane knapp
an example job w/this failure is here: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-ubuntu-testing/30/consoleFull thoughts? am i missing something obvious? i've checked and there are no avro system packages installed on any of the workers (centos or ubuntu). than

Re: need assistance debugging a strange build failure...

2018-11-05 Thread shane knapp
you tried Maven instead of SBT? This looks like a Java dependency > problem, e.g. a wrong version of Avro is picked. > > On Tue, Nov 6, 2018 at 8:30 AM shane knapp wrote: > >> i'm really close (for real: really close!) on the ubuntu port... but one >> build has be

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-04 Thread shane knapp
ild/mvn at runtime. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

[build system] jenkins wedged, not accepting new PRBs

2018-10-09 Thread shane knapp
i just restarted jenkins... hopefully this will fix the issue. of course, nothing in the logs to show why this happened (nor is there ever). shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] jenkins wedged, not accepting new PRBs

2018-10-09 Thread shane knapp
...and we're back and happily building. On Tue, Oct 9, 2018 at 3:12 PM shane knapp wrote: > i just restarted jenkins... hopefully this will fix the issue. > > of course, nothing in the logs to show why this happened (nor is there > ever). > > shane > -- > Shane

Re: moving the spark jenkins job builder repo from dbricks --> spark

2018-10-10 Thread shane knapp
/jenkins-job-builder/ On Tue, Oct 9, 2018 at 10:22 PM Sean Owen wrote: > Some responses inline -- this discussion can do to dev@ though. > > dev@ added. > On Tue, Oct 9, 2018 at 3:28 PM shane knapp wrote: > > JBB templates in spark repo: > > * code path is currently und

Re: moving the spark jenkins job builder repo from dbricks --> spark

2018-10-10 Thread shane knapp
ccess to > some test jenkins server. > > JJB has some built-in lint and testing, so that'll be the first step in verifying the build configs. i still have a dream where i have a fully functioning jenkins staging deployment... one day i will make that happen. :) shane -- Shane Knapp U

Re: Something wrong of Jenkins proxy

2018-09-24 Thread shane knapp
cert renewed and jenkins is reachable now. On Sun, Sep 23, 2018 at 8:58 PM, shane knapp wrote: > i don't manage the certs on the box doing the reverse proxy, so i've > reached out to the proper party and will hopefully things will be sorted by > early tomorrow. > > On Sun, Sep 2

[build system] jenkins wedged post-SSL cert fiasco, restarting now

2018-09-24 Thread shane knapp
jenkins will be back up and building Real Soon Now[tm]... sorry for the inconvenience! -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Something wrong of Jenkins proxy

2018-09-23 Thread shane knapp
i just noticed this... taking a look now. On Sun, Sep 23, 2018 at 4:38 AM, Yuanjian Li wrote: > Hi devs, > Is there something wrong of Jenkins proxy? > [image: image.png] > I got this proxy 500 whole days. > > Thanks, > Yuanjian Li > -- Shane Knapp UC Berkeley EECS

Re: Something wrong of Jenkins proxy

2018-09-23 Thread shane knapp
for now, you can visit: https://hadrian.ist.berkeley.edu/jenkins/ something is up w/the reverse proxy setup. On Sun, Sep 23, 2018 at 8:37 PM, shane knapp wrote: > i just noticed this... taking a look now. > > On Sun, Sep 23, 2018 at 4:38 AM, Yuanjian Li > wrote:

Re: Something wrong of Jenkins proxy

2018-09-23 Thread shane knapp
i don't manage the certs on the box doing the reverse proxy, so i've reached out to the proper party and will hopefully things will be sorted by early tomorrow. On Sun, Sep 23, 2018 at 8:37 PM, shane knapp wrote: > for now, you can visit: > > https://hadrian.ist.berkeley.ed

Re: Spark Packaging Jenkins

2019-01-06 Thread shane knapp
-- > *From:* Felix Cheung > *Sent:* Sunday, January 6, 2019 11:17 AM > *To:* Dongjoon Hyun; Wenchen Fan > *Cc:* dev; shane knapp > *Subject:* Re: Spark Packaging Jenkins > > The release process doc should have been updated on this - as mentioned we > do not use Jenk

[build system] jenkins mildly wedged, needs a quick restart

2019-01-14 Thread shane knapp
this will kill a bunch of PRB builds, so i'll go and retrigger them once jenkins is back up. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] jenkins mildly wedged, needs a quick restart

2019-01-14 Thread shane knapp
we're back up and building... things still seem a little flaky so i'll be investigating a little bit deeper in to what's doing on. On Mon, Jan 14, 2019 at 10:55 AM shane knapp wrote: > this will kill a bunch of PRB builds, so i'll go and retrigger them once > jenkins is back up. > >

Re: [build system] jenkins mildly wedged, needs a quick restart

2019-01-14 Thread shane knapp
alright, everything seems to be working as expected! :) On Mon, Jan 14, 2019 at 11:07 AM shane knapp wrote: > we're back up and building... things still seem a little flaky so i'll be > investigating a little bit deeper in to what's doing on. > > On Mon, Jan 14, 2019 at 10:55 AM

[build system] jenkins was in a mildly wedged state, just restarted

2018-12-13 Thread shane knapp
sorry if any pull request builds got killed, but jenkins was in a bad state and needed (another) full restart. looks like things are happily building now, and i can reliably log in through the UI. i'll keep an eye on things over the next couple of hours. shane -- Shane Knapp UC Berkeley EECS

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

2018-12-07 Thread shane knapp
.org, not your > project's dev list :-). > > > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

[build system] java8 updated to 8u191 on all jenkins workers

2018-12-07 Thread shane knapp
TL;DR: we're now running a 'modern' java8 on all of the workers. because of this: https://github.com/apache/spark/pull/23218 we had to do this: https://issues.apache.org/jira/browse/SPARK-26282 shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https

Re: Can I update the "2.12" PR builder to 2.11?

2018-11-20 Thread shane knapp
wrote: > The one you set up to test 2.12 separately, > spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12 > Now master is on 2.12 by default. OK will try to change it. > On Tue, Nov 20, 2018 at 10:15 AM shane knapp wrote: > > > > which build are you referring to

Re: Can I update the "2.12" PR builder to 2.11?

2018-11-20 Thread shane knapp
PR > builder to instead test 2.11. Is that just a matter of editing Jenkins > configuration that I can see and change? if so I'll just do it. > > Sean > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Can I update the "2.12" PR builder to 2.11?

2018-11-20 Thread shane knapp
-master-test-maven-hadoop-2.7-ubuntu-scala-2.12 build. On Tue, Nov 20, 2018 at 8:52 AM Sean Owen wrote: > Ah right yes not a PR builder. So would you have to update that? If > possible to get that in soon it would help detect 2.11 failures. > > On Tue, Nov 20, 2018, 10:23 AM shane

Re: Jenkins down?

2018-11-19 Thread shane knapp
lds for the PRs today. Is it in > maintenance? > > > > Thanks, > > Marco > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Jenkins down?

2018-11-19 Thread shane knapp
alright, we're back and building. On Mon, Nov 19, 2018 at 10:11 AM shane knapp wrote: > thanks for the heads up... looks like the backup process got wedged. > i'll restart jenkins now. > > On Mon, Nov 19, 2018 at 9:56 AM Sean Owen wrote: > >> Jenkins says it's shutting

Re: Make Scala 2.12 as default Scala version in Spark 3.0

2018-11-19 Thread shane knapp
in the repo manage this. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Spark Packaging Jenkins

2019-01-07 Thread shane knapp
erything becomes healthier during 2.2.3 release. > This will reduce the burden of 2.3.3/3.0.0 releases. :D > > Bests, > Dongjoon. > > > On Sun, Jan 6, 2019 at 11:42 AM Felix Cheung > wrote: > >> Awesome Shane! >> >> >> -- &

Re: Spark Packaging Jenkins

2019-01-07 Thread shane knapp
> > 1) https://issues.apache.org/jira/browse/SPARK-26565 > 2) create world-wide campaign to stop bash scripting > 3) re-create these builds w/o calling release-build.sh at all (preferred)

Re: Spark Packaging Jenkins

2019-01-04 Thread shane knapp
2) and (3) are working because it uses GitHub ( > https://github.com/apache/spark.git). > But, (1) seems to be broken because it's looking for old repo( > https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of > new GitBox. > > Can we fix this in this week? > > Bests,

Re: Spark Packaging Jenkins

2019-01-04 Thread shane knapp
wrote: > Thank you, Shane! > > Bests, > Dongjoon. > > On Fri, Jan 4, 2019 at 10:50 AM shane knapp wrote: > >> yeah, i'll get on that today. thanks for the heads up. >> >> On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun >> wrote: >> >>> Hi

Re: Spark Packaging Jenkins

2019-01-04 Thread shane knapp
https://issues.apache.org/jira/browse/SPARK-26537 On Fri, Jan 4, 2019 at 11:31 AM shane knapp wrote: > this may push in to early next week... these builds were set up before my > time, and i'm currently unraveling how they all work before pushing a > commit to fix stuff. > > no

Re: [build system] jenkins master needs reboot, temporary downtime

2018-12-19 Thread shane knapp
master is back up and building. On Wed, Dec 19, 2018 at 9:31 AM shane knapp wrote: > the jenkins process seems to be wedged again, and i think we're going to > hit it w/the reboot hammer, rather than just killing/restarting the master. > > this should take at most 30 mins, and i'll

[build system] jenkins master needs reboot, temporary downtime

2018-12-19 Thread shane knapp
the jenkins process seems to be wedged again, and i think we're going to hit it w/the reboot hammer, rather than just killing/restarting the master. this should take at most 30 mins, and i'll send an all-clear when it's done. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical

Re: moving the spark jenkins job builder repo from dbricks --> spark

2019-01-24 Thread shane knapp
above issue/PR). the maven snapshot builds are green, and working as intended (and use the ASF creds). my question is: are we currently relying on any of these doc builds? thanks in advance, shane On Wed, Oct 17, 2018 at 10:48 AM shane knapp wrote: > On Wed, Oct 17, 2018 at 10:25 AM

Re: moving the spark jenkins job builder repo from dbricks --> spark

2019-01-24 Thread shane knapp
s/dist/dev/spark/ ? I think from a thread > last month, these aren't used and should probably just be stopped. > > On Thu, Jan 24, 2019 at 3:34 PM shane knapp wrote: > > > > revisiting this thread from october... sorry for the delay in getting > around to this until now, but t

Re: Clean out https://dist.apache.org/repos/dist/dev/spark/ ?

2019-01-24 Thread shane knapp
tomorrow i will continue the purge. :) On Thu, Jan 24, 2019 at 6:13 PM Sean Owen wrote: > No and we could retire 2.2 now too but wouldn't hurt to keep it a bit > longer in case we have to make a critical release even though it's EOL. > > On Thu, Jan 24, 2019, 7:05 PM shane knapp &

Re: Clean out https://dist.apache.org/repos/dist/dev/spark/ ?

2019-01-24 Thread shane knapp
f the spark-rm docker container, but i can probably check this out tomorrow. [2] this will absolutely need to be documented somewhere (or somewheres). [3] this is my preferred solution. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Clean out https://dist.apache.org/repos/dist/dev/spark/ ?

2019-01-24 Thread shane knapp
we can just remove this job. > > On Thu, Jan 24, 2019 at 6:44 PM shane knapp wrote: > > > > On Sun, Jan 13, 2019 at 11:22 AM Felix Cheung > wrote: > >> > >> Eh, yeah, like the one with signing, I think doc build is mostly useful > when a) right befo

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-02 Thread shane knapp
e fine, as a point of > reference, pyarrow CI uses 2.7 and 3.6. > > On Mon, Apr 1, 2019 at 3:09 PM shane knapp wrote: > >> well now! color me completely surprised... i decided to whip up a fresh >> python3.6.8 conda environment this morning to "see if things just worked".

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-25 Thread shane knapp
. > > PS: I roughly heard that Shane's busy for some work stuff .. but it's kind > of important in my perspective. > > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-26 Thread shane knapp
t;> +1 on doing this in 3.0. >> >> >> On Mon, Mar 25, 2019 at 9:31 PM, Felix Cheung >> wrote: >> >>> I’m +1 if 3.0 >>> >>> >>> -- >>> *From:* Sean Owen >>> *Sent:* Monday, March

Re: [build system] jenkins wedged again, rebooting master node

2019-03-22 Thread shane knapp
quick update: since kicking httpd on the jenkins master "fixes" the GUI hanging, i set up a cron job to restart httpd 4 times per day. this is not the final solution, but will definitely help over the weekend as i'm heading out of town. shane On Fri, Mar 22, 2019 at 9:50 AM shane kn

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-29 Thread shane knapp
>> >> >> -- >> *From:* shane knapp >> *Sent:* Thursday, March 28, 2019 7:54 PM >> *To:* Hyukjin Kwon >> *Cc:* Bryan Cutler; dev; Felix Cheung >> *Subject:* Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276] &g

[k8s][jenkins] spark dev tool docs now have k8s+minikube instructions!

2019-03-28 Thread shane knapp
from hell for throwing me in the deep end of k8s. :) -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread shane knapp
> > > If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to > 3.5. > > this is inevitable. 3.4s final release was 10 days ago ( https://www.python.org/dev/peps/pep-0429/) so we're basically EOL.

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-03-28 Thread shane knapp
looks like the same for 3.5... https://www.python.org/dev/peps/pep-0478/ let's pick a python version and start testing. On Thu, Mar 28, 2019 at 7:52 PM shane knapp wrote: > >> If there was, it looks inevitable to upgrade Jenkins\s Python from 3.4 to >> 3.5. >> >>

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-01 Thread shane knapp
n any case I think in the next release it will be great to get more > Python 3.x release test coverage. > > > > ------ > *From:* shane knapp > *Sent:* Friday, March 29, 2019 4:46 PM > *To:* Bryan Cutler > *Cc:* Felix Cheung; Hyukjin Kwon; dev >

Re: Upgrading minimal PyArrow version to 0.12.x [SPARK-27276]

2019-04-01 Thread shane knapp
ever, i'll be out wednesday through friday this week and don't want to make a massive change before disappearing for a few days. so: how does early next week sound for the python upgrade? :) shane On Mon, Apr 1, 2019 at 8:58 AM shane knapp wrote: > i'd much prefer that we minimize the num

[build system] VERY IMPORTANT: please file JIRAs for issues w/jenkins

2019-03-02 Thread shane knapp
. one of my team will start helping me out w/that stuff soon, but only part-time... however, i have faith that he can bang away at the myriad parallel projects i've had going on for the past couple of years and in a few months we'll be in a much, much better spot. thanks in advance, shane -- Shane

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-06 Thread shane knapp
couple of weeks back and it worked fine. i will doubly confirm when i get in to the office today. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-06 Thread shane knapp
>> >> Does the 3.0.x client not work at all with 1.10+ or just unsupported. >> If it 'basically works but no guarantees' I'd favor not updating. If >> it doesn't work at all, hm. That's tough. I think I'd favor updating >> the client but think it's a tough call both

[build system] meet your build engineer @ the sparkAI summit!

2019-03-06 Thread shane knapp
. :) shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] jenkins pull request builds not triggering

2019-02-26 Thread shane knapp
jenkins is churning through a lot of github updates, and i'm finally seeing the backlog of pull requests builds starting. i'll keep an eye on things over the afternoon. On Tue, Feb 26, 2019 at 12:26 PM shane knapp wrote: > restarted jenkins, staring at logs. will report back when thi

[build system] jenkins pull request builds not triggering

2019-02-26 Thread shane knapp
investigating, and this will most likely require a jenkins restart. -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: PR tests not running?

2019-02-26 Thread shane knapp
bbins 于2019年2月26日周二 上午11:33写道: > >> Sorry for stating what is likely obvious, but PR tests don't appear to be >> running. Last one started was around 2AM. >> > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] jenkins pull request builds not triggering

2019-02-26 Thread shane knapp
restarted jenkins, staring at logs. will report back when things look good. On Tue, Feb 26, 2019 at 12:22 PM shane knapp wrote: > investigating, and this will most likely require a jenkins restart. > > -- > Shane Knapp > UC Berkeley EECS Research / RISELab Staff Technic

Re: [build system] jenkins wedged again, rebooting master node

2019-03-16 Thread shane knapp
On Fri, Mar 15, 2019 at 8:22 PM Sean Owen wrote: > It's not responding again. Is there any way to kick it harder? I know > it's well understood but this means not much can be merged in Spark > > it's back up and running now. btw, the only way to kick it harder would be to do a complete

Re: [build system] jenkins wedged again, rebooting master node

2019-03-16 Thread shane knapp
argh. kicking it again. On Fri, Mar 15, 2019 at 8:22 PM Sean Owen wrote: > It's not responding again. Is there any way to kick it harder? I know > it's well understood but this means not much can be merged in Spark > > On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote: > >

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread shane knapp
thanks everyone, both PRs are merged. :) On Wed, Mar 13, 2019 at 3:51 PM shane knapp wrote: > btw, let's wait and see if the non-k8s PRB tests pass before merging > https://github.com/apache/spark/pull/23993 in to 2.4.1 > > On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote: &g

[build system] jenkins wedged again, rebooting master node

2019-03-15 Thread shane knapp
spins back up. i'll reply here w/any updates. shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] jenkins wedged again, rebooting master node

2019-03-15 Thread shane knapp
well, that box rebooted in record time! we're back up and building. and as always, i'll keep a close eye on things today... jenkins usually works great, until it doesn't. :\ On Fri, Mar 15, 2019 at 9:52 AM shane knapp wrote: > as some of you may have noticed, jenkins got itself in a

Re: [build system] jenkins wedged again, rebooting master node

2019-03-17 Thread shane knapp
one > wrong-er than usual; any recent changes that could be a culprit? > > On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote: > > > > well, that box rebooted in record time! we're back up and building. > > > > and as always, i'll keep a close eye on things today...

Re: [build system] jenkins wedged again, restarted service.

2019-03-14 Thread shane knapp
things are building again! On Thu, Mar 14, 2019 at 2:02 PM shane knapp wrote: > if performance continues to drop off like this, i will reboot the box soon. > > builds should be back to normal in a few minutes... sorry for the > inconvenience! > > -- > Shane Knapp > U

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread shane knapp
nks. > > Sincerely, > > DB Tsai > -- > Web: https://www.dbtsai.com > PGP Key ID: 42E5B25A8F7A82C1 > > On Thu, Mar 14, 2019 at 9:48 AM shane knapp wrote: > > > > thanks everyone, both PRs are merged. :) &

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-14 Thread shane knapp
Mar 13, 2019 at 3:15 PM shane knapp wrote: > > > > upgrade completed, jenkins building again... master PR merged, waiting > for the 2.4.1 PR to launch the k8s integration tests. > > > > On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > >> > >> oki

[build system] jenkins wedged again, restarted service.

2019-03-14 Thread shane knapp
if performance continues to drop off like this, i will reboot the box soon. builds should be back to normal in a few minutes... sorry for the inconvenience! -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
ok awesome. let's shoot for 3pm PST. On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin wrote: > On Wed, Mar 13, 2019 at 11:53 AM shane knapp wrote: > > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin > wrote: > >> > >> Do the upgraded minikube/k8s versions b

[discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
, shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
8s integration tests on branch-2.4 for a > little bit, that would be optimal IMO. > > i have everything staged on the affected jenkins workers and can do the infra upgrade really quickly... shane > On Wed, Mar 13, 2019 at 11:26 AM shane knapp wrote: > > > > hey everyone..

Re: [build system] jenkins wedged again, rebooting master node

2019-03-18 Thread shane knapp
means not much can be merged in Spark > > On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote: > > > > well, that box rebooted in record time! we're back up and building. > > > > and as always, i'll keep a close eye on things today... jenkins usually > works great, un

[build system] jenkins web UI wedged, restarting service

2019-03-12 Thread shane knapp
i'm getting timeouts trying to connect to the site, so i'm preemptively going to kick the service. should be back up and running really soon. i'll reply here w/any updates. shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] jenkins web UI wedged, restarting service

2019-03-12 Thread shane knapp
alright, jenkins is back up and the UI is super snappy! sorry for the interruption in service. On Tue, Mar 12, 2019 at 10:51 AM shane knapp wrote: > i'm getting timeouts trying to connect to the site, so i'm preemptively > going to kick the service. should be back up and running reall

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
okie dokie! the time approacheth! i'll pause jenkins @ 3pm to not accept new jobs. i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds. On Wed, Mar 13, 2019 at 12:17 PM shane knapp wrote: > ok awesome. let's shoot for 3pm PST. > > On We

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
2.4.1 k8s integration test passed: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/ thanks everyone! :) On Wed, Mar 13, 2019 at 3:24 PM shane knapp wrote: > 2.4.1 integration tests running: > https://amplab.cs.berkeley.edu/jenkins/job/t

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-07 Thread shane knapp
https://issues.apache.org/jira/browse/SPARK-26742 On Thu, Mar 7, 2019 at 10:52 AM shane knapp wrote: > i'm ready to update the ubuntu workers/minikube/k8s to support the 4.1.2 > client: > https://issues.apache.org/jira/browse/SPARK-2674 > > i am more than comfortable with th

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

2019-03-07 Thread shane knapp
the 3.0.x client not work at all with 1.10+ or just unsupported. >> If it 'basically works but no guarantees' I'd favor not updating. If >> it doesn't work at all, hm. That's tough. I think I'd favor updating >> the client but think it's a tough call both ways. >> >&g

Re: [build system] jenkins wedged again, rebooting master node

2019-03-21 Thread shane knapp
, however... only time will tell. On Tue, Mar 19, 2019 at 7:18 AM Imran Rashid wrote: > seems wedged again? > > sorry for the bad news Shane, thanks for all the work on fixing it > > On Mon, Mar 18, 2019 at 4:02 PM shane knapp wrote: > >> ok, i dug through the logs a

Re: [build system] jenkins wedged again, rebooting master node

2019-03-22 Thread shane knapp
again, shane On Thu, Mar 21, 2019 at 12:54 PM shane knapp wrote: > i tweaked some apache settings (MaxClients increased to fix an error i > found buried in the logs, and added 'retry' and 'acquire' to the reverse > proxy settings to hopefully combat the dreaded 502 response), restarte

Re: merge script stopped working; Python 2/3 input() issue?

2019-02-15 Thread shane knapp
On Fri, Feb 15, 2019 at 4:03 PM Reynold Xin wrote: > lol > > couldn't have said it better myself. ;) shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Jenkins commands?

2019-02-06 Thread shane knapp
k runs in the pull request builds and the > QA builds? > > > Thanks, > Tom > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] Jenkins stopped working

2019-02-19 Thread shane knapp
8:29 AM Hyukjin Kwon wrote: > >> Hi all, >> >> Looks Jenkins stopped working. Did I maybe miss a thread, or anybody >> didn't report this yet? >> >> Thanks! >> >> >> -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [build system] Jenkins stopped working

2019-02-19 Thread shane knapp
forgot to hit send before i went in to the office: we're back up and building! On Tue, Feb 19, 2019 at 8:06 AM shane knapp wrote: > yep, it got wedged. issued a restart and it should be back up in a few > minutes. > > On Tue, Feb 19, 2019 at 7:32 AM Parth Gandhi > wrote: >

Re: [build system] Jenkins stopped working

2019-02-19 Thread shane knapp
aand i had to issue another restart. it's the ever annoying, and never quite clear as to why it's happening proxy/502 error. currently investigating. On Tue, Feb 19, 2019 at 9:21 AM shane knapp wrote: > forgot to hit send before i went in to the office: we're back up and > bu

Re: [build system] Jenkins stopped working

2019-02-19 Thread shane knapp
alright, i increased the httpd and proxy timeouts and kicked apache. i'll keep an eye on things, but as of right now we're happily building. On Tue, Feb 19, 2019 at 2:25 PM shane knapp wrote: > aand i had to issue another restart. it's the ever annoying, and > never quite clear as

Re: Welcome Jose Torres as a Spark committer

2019-01-29 Thread shane knapp
Jose Torres as a committer on the >>>> project. Jose has been a major contributor to Structured Streaming. Please >>>> join me in welcoming him! >>>> >>>> Best Regards, >>>> >>>> Shixiong Zhu >>>> >>>> -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Clean out https://dist.apache.org/repos/dist/dev/spark/ ?

2019-01-25 Thread shane knapp
s): https://issues.apache.org/jira/browse/SPARK-26608 shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: Clean out https://dist.apache.org/repos/dist/dev/spark/ ?

2019-01-25 Thread shane knapp
https://issues.apache.org/jira/browse/SPARK-26731 On Fri, Jan 25, 2019 at 10:27 AM shane knapp wrote: > On Thu, Jan 24, 2019 at 6:13 PM Sean Owen wrote: > >> No and we could retire 2.2 now too but wouldn't hurt to keep it a bit >> longer in case we have to make a critical r

[build system] jenkins temporarily unreachable, fixing now

2019-04-09 Thread shane knapp
the host serving the reverse proxy to jenkins is wedged and is being rebooted now. jenkins is still up and happily building, so no jenkins restart required. shane -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

2019-04-10 Thread shane knapp
3.4; would this mean it's now just > tested vs 3.6? that's not out of the question, but can the older > branches continue to test on older versions or is that super complex? > > On Wed, Apr 10, 2019 at 1:37 PM shane knapp wrote: > > > > details here (see most recent comments

Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

2019-04-15 Thread shane knapp
is still 'supported' > in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway. > This could be fine if we accept that small risk and it keeps the > testing env from getting crazy; I don't actually want to test vs N > Python versions anyway. > > On Wed, Apr 10, 2019 at

Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

2019-04-15 Thread shane knapp
, Apr 15, 2019 at 11:49 AM shane knapp wrote: > 1) i absolutely do not want to test against more than two python > versions. consider my foot to have been put down on that. :) > > 2) i'll start testing against 2.3 and 2.4 now (last week was a bit crazy, > so i didn't get around t

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

2019-03-13 Thread shane knapp
upgrade completed, jenkins building again... master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests. On Wed, Mar 13, 2019 at 2:55 PM shane knapp wrote: > okie dokie! the time approacheth! > > i'll pause jenkins @ 3pm to not accept new jobs. i don

<    1   2   3   4   5   6   7   8   >