2.4.1 integration tests running:
https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
On Wed, Mar 13, 2019 at 3:15 PM shane knapp wrote:
> upgrade completed, jenkins building again... master PR merged, waiting
> for the 2.4.1 PR to launch t
btw, let's wait and see if the non-k8s PRB tests pass before merging
https://github.com/apache/spark/pull/23993 in to 2.4.1
On Wed, Mar 13, 2019 at 3:42 PM shane knapp wrote:
> 2.4.1 k8s integration test passed:
>
>
> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-pr
machines are down, gpus are about to go in. i expect these workers to back
up and building in ~30min.
On Fri, May 17, 2019 at 1:47 PM shane knapp wrote:
> we're installing some new GPUs for builds to use for tests... the
> following workers will be offline for the next couple of
for both lab and spark integration tests, but will
NOT BE READY for the next couple of weeks.
i repeat: even though there will be GPUs, they will not be ready for use
yet. ;)
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
amp-jenkins-staging-worker-02 and ubuntu-testing are back up.
-01 is being a little reluctant to boot and we're investigating.
On Fri, May 17, 2019 at 2:08 PM shane knapp wrote:
> machines are down, gpus are about to go in. i expect these workers to
> back up and building in
;>>>> On Thu, May 30, 2019 at 2:18 AM Felix Cheung <
>>>>> felixcheun...@hotmail.com> wrote:
>>>>>
>>>>>> We don’t usually reference a future release on website
>>>>>>
>>>>>> > Spark website and
't be expanding.
that being said, i will be cracking a bottle of champagne when i can delete
all of the ansible and anaconda configs for python2.x. :)
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
scuss before
>>>> making a change. Does anyone else have thoughts on this?
>>>>
>>>> Regards,
>>>> Bryan
>>>>
>>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
everything back up on sunday, august 4th. if they run in to issues, i will
jump in first thing monday, august 5th.
as the time approaches, i will send reminders and updates.
thanks,
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
ative, so it's pretty unlikely that we
> will add something that would break 0.23.X.
>
> On Fri, Jun 14, 2019 at 10:10 AM shane knapp wrote:
>
>> ah, ok... should we downgrade the testing env on jenkins then? any
>> specific version?
>>
>> shane, who is lo
ear ago, 0.24.0 in
> January and 0.24.2 in March.
>
> On Fri, Jun 14, 2019 at 9:27 AM shane knapp wrote:
>
>> just to everyone knows, our python 3.6 testing infra is currently on
>> 0.24.2...
>>
>> On Fri, Jun 14, 2019 at 9:16 AM Dongjoon Hyun
>> wrote:
>>
On Fri, Jun 14, 2019 at 11:36 AM Felix Cheung
> wrote:
>
>> How about pyArrow?
>>
>> --
>> *From:* Holden Karau
>> *Sent:* Friday, June 14, 2019 11:06:15 AM
>> *To:* Felix Cheung
>> *Cc:* Bryan Cutler; Dongjoon
i will do it later this week. also, in the future, please file jiras for
stuff like this rather than pinging me on the list. ;)
On Wed, Jun 19, 2019 at 1:39 PM Xiao Li wrote:
> That sounds good to me!
>
> @shane knapp Could you help this? Or Dongjoon can
> do it by himself
i'm aware and will be looking in to this later today.
see:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/6043/console
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
ah, ok. thanks for letting me know. :)
On Mon, Jun 24, 2019 at 9:39 AM Sean Owen wrote:
> (We have two PRs to patch it up anyway already)
>
> On Mon, Jun 24, 2019 at 11:39 AM shane knapp wrote:
> >
> > i'm aware and will be looking in to this later today.
>
all workers are now up, online and ready to build!
On Fri, May 17, 2019 at 2:55 PM shane knapp wrote:
> amp-jenkins-staging-worker-02 and ubuntu-testing are back up.
>
> -01 is being a little reluctant to boot and we're investigating.
>
> On Fri, May 17, 2019 at 2:08 PM sha
actually, amp-jenkins-staging-worker-01 is seriously unhappy and just
crashed. we will investigate more on monday.
:(
shane
On Fri, May 17, 2019 at 3:19 PM shane knapp wrote:
> all workers are now up, online and ready to build!
>
> On Fri, May 17, 2019 at 2:55 PM shane knapp wrote:
>>>>> like we lose much information.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd close anything that hasn't had activity in 2 years (?), as
>>>>>>>>>>>> a start.
>>>>>>>>>>>> I like the idea of closing things that only affect an EOL
>>>>>>>>>>>> release,
>>>>>>>>>>>> but, many items aren't marked, so may need to cast the net
>>>>>>>>>>>> wider.
>>>>>>>>>>>>
>>>>>>>>>>>> I think only then does it make sense to look at bothering to
>>>>>>>>>>>> reproduce
>>>>>>>>>>>> or evaluate the 1000s that will still remain.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, May 15, 2019 at 4:25 AM Hyukjin Kwon <
>>>>>>>>>>>> gurwls...@gmail.com> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > Hi all,
>>>>>>>>>>>> >
>>>>>>>>>>>> > I would like to propose to resolve all JIRAs that affects EOL
>>>>>>>>>>>> releases - 2.2 and below. and affected version
>>>>>>>>>>>> > not specified. I was rather against this way and considered
>>>>>>>>>>>> this as last resort in roughly 3 years ago
>>>>>>>>>>>> > when we discussed. Now I think we should go ahead with this.
>>>>>>>>>>>> See below.
>>>>>>>>>>>> >
>>>>>>>>>>>> > I have been talking care of this for so long time almost
>>>>>>>>>>>> every day those 3 years. The number of JIRAs
>>>>>>>>>>>> > keeps increasing and it does never go down. Now the number is
>>>>>>>>>>>> going over 2500 JIRAs.
>>>>>>>>>>>> > Did you guys know? in JIRA, we can only go through page by
>>>>>>>>>>>> page up to 1000 items. So, currently we're even
>>>>>>>>>>>> > having difficulties to go through every JIRA. We should
>>>>>>>>>>>> manually filter out and check each.
>>>>>>>>>>>> > The number is going over the manageable size.
>>>>>>>>>>>> >
>>>>>>>>>>>> > I am not suggesting this without anything actually trying.
>>>>>>>>>>>> This is what we have tried within my visibility:
>>>>>>>>>>>> >
>>>>>>>>>>>> > 1. In roughly 3 years ago, Sean tried to gather committers
>>>>>>>>>>>> and even non-committers people to sort
>>>>>>>>>>>> > out this number. At that time, we were only able to keep
>>>>>>>>>>>> this number as is. After we lost this momentum,
>>>>>>>>>>>> > it kept increasing back.
>>>>>>>>>>>> > 2. At least I scanned _all_ the previous JIRAs at least
>>>>>>>>>>>> more than two times and resolved them. Roughly
>>>>>>>>>>>> > once a year. The rest of them are mostly obsolete but not
>>>>>>>>>>>> enough information to investigate further.
>>>>>>>>>>>> > 3. I strictly stick to "Contributing to JIRA Maintenance"
>>>>>>>>>>>> https://spark.apache.org/contributing.html and
>>>>>>>>>>>> > resolve JIRAs.
>>>>>>>>>>>> > 4. Promoting other people to comment on JIRA or actively
>>>>>>>>>>>> resolve them.
>>>>>>>>>>>> >
>>>>>>>>>>>> > One of the facts I realised is the increasing number of
>>>>>>>>>>>> committers doesn't virtually help this much (although
>>>>>>>>>>>> > it might be helpful if somebody active in JIRA becomes a
>>>>>>>>>>>> committer.)
>>>>>>>>>>>> >
>>>>>>>>>>>> > One of the important thing I should note is that, it's now
>>>>>>>>>>>> almost pretty difficult to reproduce and test the
>>>>>>>>>>>> > issues found in EOL releases. We should git clone, checkout,
>>>>>>>>>>>> build and test. And then, see if that issue
>>>>>>>>>>>> > still exists in upstream, and fix. This is non-trivial
>>>>>>>>>>>> overhead.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Therefore, I would like to propose resolving _all_ the JIRAs
>>>>>>>>>>>> that targets EOL releases - 2.2 and below.
>>>>>>>>>>>> > Please let me know if anyone has some concerns or objections.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -
>>>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
and i really expect things to be
back up and running pretty quickly. i will send updates as needed.
shane
1-- this will be for 2.3/2.4 only, and tests against pandas 0.19.2 and
pyarrow 0.8.0. master tests against pandas 0.23.2 and pyarrow 0.12.1
--
Shane Knapp
UC Berkeley EECS Research / RISELab
this afternoon about the upgrade.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
i will actually be doing this now!
On Thu, Apr 18, 2019 at 2:57 PM shane knapp wrote:
> well, upon us on monday. :)
>
> firstly, an important note: if you have an open PR, please check to see
> if you need to rebase my changes on it before testing.
>
> monday @ 11am
hould
>> be worked on immediately. Everything else please retarget to an
>> appropriate release.
>>
>> ======
>> But my bug isn't fixed?
>> ==
>>
>> In order to make timely releases, we will typically not hold the
>> release unless the bug in question is a regression from the previous
>> release. That being said, if there is something which is a regression
>> that has not been correctly targeted please ping me or a committer to
>> help target the issue.
>>
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
and this is done.
welcome to the brave new world of python3.6!
On Fri, Apr 19, 2019 at 9:34 AM shane knapp wrote:
> i will actually be doing this now!
>
>
>
> On Thu, Apr 18, 2019 at 2:57 PM shane knapp wrote:
>
>> well, upon us on monday. :)
>>
>> firs
jenkins is back up, and PRB jobs should start trickling in.
On Fri, Apr 26, 2019 at 12:20 PM shane knapp wrote:
> if this doesn't fix things, i will reboot the server.
>
> shane (who is also dealing w/a major sewer backup @ his house)
> --
> Shane Knapp
> UC Berkeley EECS
if this doesn't fix things, i will reboot the server.
shane (who is also dealing w/a major sewer backup @ his house)
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
(schedule TBD).
our booth is right behind pepperdata's, on the left as you enter the expo
floor.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
/apache/spark/pull/24266
question: which other branches should i be applying this change to?
definitely master and 2.4... also 2.3? 2.2?
thanks in advance,
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
to have the ARM CI to make sure the
>>> spark is compatible for AArch64 platforms.
>>> >>
>>> >> I’m from OpenLab team(https://openlabtesting.org/ ,a community to do
>>> open source project testing. And we can support some Arm virtual machines
>>
...or via VM as you mentioned earlier. :)
shane (who will file a JIRA tomorrow)
On Tue, Jun 25, 2019 at 6:44 PM shane knapp wrote:
> i'd much prefer that we keep the test/build infrastructure in one place.
>
> we don't have ARM hardware, but there's a slim possibility i can scare
&g
plit them and
>> the details about different CI jobs does? Such as PR title contains [SQL],
>> [INFRA], [ML], [DOC], [CORE], [PYTHON], [k8s], [DSTREAMS], [MLlib],
>> [SCHEDULER], [SS],[YARN], [BUIILD] and etc..I found each of them seems run
>> the different CI job
i kicked apache, and the number of active connections dropped from ~500 to
~100... things are much snappier now.
On Fri, Aug 16, 2019 at 1:05 PM Shane Knapp wrote:
> it's that time of the year again when jenkins decides to take it's time
> serving pages.
>
> i will post any update
PM Shane Knapp wrote:
> quick update:
>
> it's been 4 hours, the colo is still down, and i haven't gotten any news
> yet as to when they're planning on getting power restored.
>
> once i hear something i will let everyone know what's up.
>
> On Wed, Aug 14, 2019 at 10:
quick update:
it's been 4 hours, the colo is still down, and i haven't gotten any news
yet as to when they're planning on getting power restored.
once i hear something i will let everyone know what's up.
On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp wrote:
> the berkeley colo had a major po
a couple of workers needed a bit more time to finish booting up, so no need
for my excursion tomorrow. :)
builds be building, things look happy.
On Thu, Aug 15, 2019 at 6:46 PM Shane Knapp wrote:
> it's back up! some of the workers didn't come back cleanly, so i'll have
> to hit up th
it's back up! some of the workers didn't come back cleanly, so i'll have
to hit up the colo tomorrow and persuade them in person.
On Thu, Aug 15, 2019 at 6:45 PM Wenchen Fan wrote:
> Thanks for tracking it Shane!
>
> On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp wrote:
>
>>
a PR I'm not able to see
> if "Test build" is being processed. But I can see this after a few minutes
> or hours later. Is it possible to see if "Test Build" is being processed
> after PR is created right away?
>
> Thank you,
> Younggyu Chun
>
--
Shane Knapp
U
for the inconvenience...
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
it's that time of the year again when jenkins decides to take it's time
serving pages.
i will post any updates/etc when i get them.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
ny many contributions,
>>>>>>> Apache Spark master branch starts to pass on JDK11 as of today.
>>>>>>> (with `hadoop-3.2` profile: Apache Hadoop 3.2 and Hive 2.3.6)
>>>>>>>
>>>>>>>
>>>>>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/326/
>>>>>>> (JDK11 is used for building and testing.)
>>>>>>>
>>>>>>> We already verified all UTs (including PySpark/SparkR) before.
>>>>>>>
>>>>>>> Please feel free to use JDK11 in order to build/test/run `master`
>>>>>>> branch and
>>>>>>> share your experience including any issues. It will help Apache
>>>>>>> Spark 3.0.0 release.
>>>>>>>
>>>>>>> For the follow-ups, please follow
>>>>>>> https://issues.apache.org/jira/browse/SPARK-24417 .
>>>>>>> The next step is `how to support JDK8/JDK11 together in a single
>>>>>>> artifact`.
>>>>>>>
>>>>>>> Bests,
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> [image: Databricks Summit - Watch the talks]
>>>>> <https://databricks.com/sparkaisummit/north-america>
>>>>>
>>>>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
gt; In theory we also have read/write access to github.com now too, but
> right now it hadn't yet worked for me. It may need to sync. This note
> just makes sure anyone knows how to keep pushing commits right now to
> the new ASF repo.
>
> Report any problems here!
>
> Sean
>
&
, and why we could
> probably collapse some too.
>
> Comments welcome on the JIRA, as I'm sure I'm missing a thing or two.
i will definitely be adding my thoughts, but i most likely won't be
able to get to this until after the labor day holiday (i'm busy
writing performance reviews).
shane
--
S
ate, we had a
> Mid-Autumn holiday:)
>
> If you hope to integrate ARM CI to amplab jenkins, we can offer the arm
> instance, and then the ARM job will run together with other x86 jobs, so
> maybe there is a guideline to do this? @shane knapp
> would you help us?
>
> On Thu, S
EOM
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
it's back up and building.
On Mon, Sep 16, 2019 at 11:02 AM Shane Knapp wrote:
>
> EOM
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff
what i CAN do first thing tomorrow is pause builds, wipe out the
ivy/sbt caches and SparkPullRequestBuilder* dirs on all workers and
see if that helps.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.
it helps.
>
> On Tue, Sep 17, 2019 at 1:08 PM Shane Knapp wrote:
>>
>> > Note that they're different workers (amp-jenkins-worker-0x) with same
>> > workspace /home/jenkins/workspace/SparkPullRequestBuilder@6.
>> >
>> the '@6' refers to the executor n
l the best,
>
> Matei and the Spark PMC
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
that i manually cancel.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
d; can you just delete ~/.m2 and let it download the
> internet again? or at least blow away the downloaded Kafka dir?
> Turning it on and off, so to speak, often works.
>
> On Tue, Sep 17, 2019 at 2:41 PM Shane Knapp wrote:
> >
> > a bunch of the PRB builds are now
> <--- not good!
i rmed that file and hopefully it will repopulate w/o issue. if not,
i'll kill builds on that worker and wipe all local caches. again.
On Tue, Sep 17, 2019 at 12:53 PM Shane Knapp wrote:
>
> that's what i literally just did! i wiped the .m2, .ivy and
> per-executor sb
this is done and jenkins is building again!
On Tue, Sep 17, 2019 at 10:14 AM Shane Knapp wrote:
>
> i'm going to clean up the spark workspaces on the jenkins workers and
> clear out ivy and maven caches.
>
> this means no new builds will be started as of right now, and current
>
rry for the terminal font setup, but there are no erroneous control
characters popping up, and -e shows a $ at EOL)
i'm confused that this is happening. anyone have any ideas?
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.
> ah, i found this sucker on amp-jenkins-worker-02:
s/02/06
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
On Fri, Jun 14, 2019 at 9:13 AM shane knapp wrote:
> the campus colo will be performing some electrical maintenance, which
> means that they'll be powering off the entire building.
>
> since the jenkins cluster is located in that colo, we are most definitely
> affected. :)
here's the latest timetable:
* all machines powered off some time tomorrow (friday) night ~9pm
* sunday morning, all machines will be powered back up
* if any stragglers fail to come back, we will investigate monday morning
On Tue, Jul 30, 2019 at 11:30 AM shane knapp wrote:
> On Fri, Jun
of builds.
as a temporary work-around, i will see if i can find a cached version of
pycodestyle.py somewhere and put that in each worker's PATH.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
ok, it seems to be back now, but github is experiencing major outages right
now:
https://www.githubstatus.com/
shane
On Mon, Jul 22, 2019 at 9:01 AM shane knapp wrote:
> hey everyone!
>
> i was debugging some SSL cert issues this morning, and noticed that once i
> got that resol
apologize for any inconvenience that this will cause... and i hope
that this is the final piece of the power puzzle that our colo team
has to deal with.
thanks in advance,
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
aand that was quick! everything is back up and building
On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp wrote:
>
> power switchover is happening now. more updates to come once machines
> come back up.
>
> On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp wrote:
> >
> > t
power switchover is happening now. more updates to come once machines
come back up.
On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp wrote:
>
> the main transformer for our colo is experiencing major issues, and
> campus be performing emergency work on it starting tomorrow morning
> (tue
9 at 6:07 AM Jungtaek Lim wrote:
>>
>> Awesome, thanks for the quick update!
>>
>> On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp wrote:
>>>
>>> no worries. since we deprecated the packaging builds i put that
>>> worker back in to the rotation...
,
shane
On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp wrote:
>
> aand that was quick! everything is back up and building
>
> On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp wrote:
> >
> > power switchover is happening now. more updates to come once machines
> > com
about dropping 2.7 so quickly.
thoughts? opinions?
On Thu, Oct 31, 2019 at 1:00 PM Shane Knapp wrote:
> i'm currently testing PyPy3.6 v7.2.0 w/this pull request:
> https://github.com/apache/spark/pull/26330
>
> On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz
> wrote:
>
&
gt;>> can also be removed at the same time. There are also other ways to create
>>> Rows that will not be affected. I have opened a JIRA [3] to capture this,
>>> but I am wondering what others think about fixing this for Spark 3.0?
>>>
>>> [1] https://github.com/apache/spark/pull/20280
>>> [2] https://www.python.org/dev/peps/pep-0468/
>>> [3] https://issues.apache.org/jira/browse/SPARK-29748
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
i'm currently testing PyPy3.6 v7.2.0 w/this pull request:
https://github.com/apache/spark/pull/26330
On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz
wrote:
> Could we upgrade to PyPy3.6 v7.2.0?
> On 10/30/19 9:45 PM, Shane Knapp wrote:
>
> one quick thing: we currently
also, here's my PR for dropping 2.7 tests:
https://github.com/apache/spark/pull/26330
On Wed, Oct 30, 2019 at 1:45 PM Shane Knapp wrote:
> one quick thing: we currently test against python2.7, 3.6 *and* pypy2.5.1
> (python2.7).
>
> what are our plans for pypy?
>
>
> On We
2019 at 5:45 AM Takeshi Yamamuro
> wrote:
>
>> +1, too.
>>
>> On Tue, Oct 29, 2019 at 4:16 PM Holden Karau
>> wrote:
>>
>>> +1 to deprecating but not yet removing support for 3.6
>>>
>>> On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp
sure. that shouldn't be too hard, but we've historically given very little
support to it.
On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz
wrote:
> Could we upgrade to PyPy3.6 v7.2.0?
> On 10/30/19 9:45 PM, Shane Knapp wrote:
>
> one quick thing: we currently test against py
it's coming back up now.
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
on: No suitable driver found for
> >> jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> >>
> >>
> >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class
> >> org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not
> >> org.apache.hadoop.hive.metastore.MetaStoreFilterHook
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
FYI, i just filed https://issues.apache.org/jira/browse/SPARK-30111
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr
CURRENT_BLOCK, so
> # the Jenkins scripts will report all failures as "some tests failed"
> rather than a more specific
> # error message.
> if [ ! -f "dev/run-tests-jenkins" ]; then
> wget
> https://raw.githubusercontent.com/apache/spark/master/dev/ru
https://github.com/apache/spark/pull/26753
ilan, thanks for taking care of this!
the k8s prb and master test are green now, and the jdk11 build failed
for an unrelated reason.
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-k8s-jdk11/
On Tue, Dec 3, 2019 at 10:21 AM Shane Knapp
it to return in
~3.5 hours.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
i will need to restart jenkins -- the worker's ENV vars got borked when
they came back up.
this is happening NOW.
shane
On Mon, Oct 28, 2019 at 10:37 AM Shane Knapp wrote:
> we're back up and building!
>
> On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp wrote:
>
>> ok, it l
ecifically, can we start to deprecate JDK8u81 and older at 3.0.
>
> Currently, Apache Spark testing infra are testing only with jdk1.8.0_191 and
> above.
>
> Bests,
> Dongjoon.
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff T
ion.
>>>>
>>>> Specifically:
>>>>
>>>>- Python 3.4 has been retired this year.
>>>>- Python 3.5 is already in the "security fixes only" mode and
>>>>should be retired in the middle of 2020.
>>>>
>>>> Continued support of these two blocks adoption of many new Python
>>>> features (PEP 468) and it is hard to justify beyond 2020.
>>>>
>>>> Should these two be deprecated in 3.0.0 as well?
>>>>
>>>> --
>>>> Best regards,
>>>> Maciej
>>>>
>>>>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
we're back up and building!
On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp wrote:
> ok, it looks like the colo will have power until monday morning, and
>> it will be shut down from 8am to noon to perform some maintenance.
>>
>> this means jenkins will be up all weekend, bu
workers will shut down
> and power back on automatically when power is restored.
>
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.
this means jenkins will be up all weekend, but down monday morning.
more information for either of these events.
the situation regarding the power shutoff is fluid and changing, and
depending on the weather and winds, could potentially not happen at
all, or impact us until monday.
once i hear more i will update this message.
shane
--
Shane Knapp
UC Berkeley EECS
, so if your PR or
'retest this please' comment doesn't seem to catch and trigger a build
after ~15 mins, feel free (if you're whitelisted or an admin) to
request another test with 'test this please'.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.
remove the legacy
> > Jenkins jobs via the following JIRA issue.
> >
> > https://issues.apache.org/jira/browse/SPARK-29935
> >
> > Please let me know if you have any concerns on this.
> > (We can keep the legacy jobs, but two of them are already broken.)
> >
> > Bests,
> > Dongjoon.
&g
i had a few minutes and everything has been deleted!
On Tue, Nov 19, 2019 at 2:02 PM Shane Knapp wrote:
>
> thank sean!
>
> i am all for moving these jobs to github actions, and will be doing
> this 'soon' as i'm @ kubecon this week.
>
> btw the R ecosystem definitely
ted.
>
>
> Could you please check on this?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp wrote:
>>
>> quick update from our colo admin: they are going to keep the colo on
>> generator power until monday morning and not s
of time
for these builds to run... do we need to increase it? can
someone(s) here figure out what's taking so long and refactor some of
the tests?
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
just chatted w/sean privately and i'm going to up the test timeouts to
480mins (8 hours).
i still don't like this but at least it should hopefully get things green again.
On Mon, Oct 7, 2019 at 11:31 AM Shane Knapp wrote:
>
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test
s to come back up, manually run things ourselves and commit?
>>>>
>>>> Tom
>>>>
>>>> -
>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>>
> --
t’ll be a
>> slower process but I don’t think we need to halt all merging.
>>
>> On Thu, Oct 10, 2019 at 9:07 AM Shane Knapp wrote:
>>>
>>> if we do get power back before the weekend, i can have my sysadmin
>>> head down to the colo friday afternoon and
1写道：
>>>
>>> Awesome, thanks Shane :)
>>>
>>> In the meantime I think committers can just run tests locally and it’ll be
>>> a slower process but I don’t think we need to halt all merging.
>>>
>>> On Thu, Oct 10, 2019 at 9:07 AM Shan
another quick update:
campus lost power ~1130pm, and is closed for the entirety of today.
no word on power restoration, campus status, etc etc.
updates as they come. :\
On Wed, Oct 9, 2019 at 2:34 PM Shane Knapp wrote:
>
> quick update:
>
> campus is losing power @ 8pm. this
load balancers. :\
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
service will be restored (i HOPE) by friday morning.
shane (who doesn't ever want to check this list's archives and count
how many times we've had power issues)
On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp wrote:
>
> here in the lovely bay area, we are currently experiencing some
> absolute
jenkins is going down now.
On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp wrote:
>
> quick update:
>
> we are definitely going to have our power shut off starting early
> tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> before it is restored.
>
> i will
jenkins) until monday.
more updates as they come. thanks for your patience!
On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp wrote:
>
> jenkins is going down now.
>
> On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp wrote:
> >
> > quick update:
> >
> > we are definit
it was wedged and i had to perform a quick restart. sorry about the
interruption of service!
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
-
To unsubscribe e-mail
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-maven-snapshots/
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-maven-snapshots/
i created dry-run test builds and everything looked great. please
file a JIRA is anything published by these jobs looks fishy.
shane
--
Shane
. this means any open PRs will need to be retriggered.
please don't do this until *after* you get the notice that jenkins is
up.
thanks for your patience,
shane
On Thu, Oct 10, 2019 at 9:05 AM Shane Knapp wrote:
>
> another quick update:
>
> campus lost power ~1130pm, and is closed for
thanks jon!
On Fri, Oct 11, 2019 at 3:25 PM Jon Kuroda wrote:
>
> Jenkins is back up and building. Build queue purged, builders are building.
> Fiat Lux --Jon
>
> On Fri, Oct 11, 2019 at 1:58 PM Shane Knapp wrote:
>>
>> finally, some good news! power was just re
Kwon wrote:
> Hi all,
>
> I just noticed we apparently don't build the documentation in the Jenkins
> anymore.
> I remember we have the job:
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-docs
> Does anybody know what happened to this job?
>
> Thanks.
>
cumentation (
>>> https://spark.apache.org/docs/latest/api/sql/index.html) are
>>> not being tested anymore if I am not mistaken. I believe
>>> spark-master-docs
>>> <https://amplab.cs.berkeley.edu/jenkins/job/spark-master-docs> was only
>>>
501 - 600 of 720 matches
Mail list logo
