Re: [build system] jenkins temporarily unreachable, fixing now

[William Wong]

Hi,

I could not access Jenkins via url
https://amplab.cs.berkeley.edu/ anymore.

May I know if the reverse proxy is still having issue? Many thanks

Best regards,
William

On Wed, 10 Apr 2019 at 2:34 AM, shane knapp  wrote:

> the host serving the reverse proxy to jenkins is wedged and is being
> rebooted now.  jenkins is still up and happily building, so no jenkins
> restart required.
>
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>

Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

[Sean Owen]

My guess is that branch 2.3, 2.4 still work with Python 3.6 even if
we're not testing that. If we're only testing against 3.6, I guess we
wouldn't detect if something broke for 3.4 which is still 'supported'
in 2.3.x, 2.4.x? For 3.x, no problem, let's raise the minimum anyway.
This could be fine if we accept that small risk and it keeps the
testing env from getting crazy; I don't actually want to test vs N
Python versions anyway.

On Wed, Apr 10, 2019 at 4:49 PM shane knapp  wrote:
>
> ok, i'll create a PR for each branch so we can test things that way, rather 
> than backporting.
>
> the bigger question i have is how far back, branch-wise, are we willing to 
> support w/regards to tests?
>
> On Wed, Apr 10, 2019 at 12:16 PM Sean Owen  wrote:
>>
>> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
>> tested vs 3.6? that's not out of the question, but can the older
>> branches continue to test on older versions or is that super complex?
>>
>> On Wed, Apr 10, 2019 at 1:37 PM shane knapp  wrote:
>> >
>> > details here (see most recent comments for current state of things):
>> > https://issues.apache.org/jira/browse/SPARK-25079
>> >
>> > my PR for these changes:
>> > https://github.com/apache/spark/pull/24266
>> >
>> > we're doing this because we need to support arrow 0.12.1:
>> > https://issues.apache.org/jira/browse/SPARK-27276
>> >
>> > a couple of things:
>> >
>> > * i won't be switching things over until april 11th or 12th at the 
>> > earliest.
>> > *  this change will require a short (~15-20min) downtime to switch over 
>> > from 3.4 to 3.6.
>> > *  updating python will impact all active branches, so i will need to 
>> > backport this pr:  https://github.com/apache/spark/pull/24266
>> >
>> > question:  which other branches should i be applying this change to?  
>> > definitely master and 2.4...  also 2.3?  2.2?
>> >
>> > thanks in advance,
>> >
>> > shane
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

[shane knapp]

ok, i'll create a PR for each branch so we can test things that way, rather
than backporting.

the bigger question i have is how far back, branch-wise, are we willing to
support w/regards to tests?

On Wed, Apr 10, 2019 at 12:16 PM Sean Owen  wrote:

> In theory Spark 2.4 supports Python 3.4; would this mean it's now just
> tested vs 3.6? that's not out of the question, but can the older
> branches continue to test on older versions or is that super complex?
>
> On Wed, Apr 10, 2019 at 1:37 PM shane knapp  wrote:
> >
> > details here (see most recent comments for current state of things):
> > https://issues.apache.org/jira/browse/SPARK-25079
> >
> > my PR for these changes:
> > https://github.com/apache/spark/pull/24266
> >
> > we're doing this because we need to support arrow 0.12.1:
> > https://issues.apache.org/jira/browse/SPARK-27276
> >
> > a couple of things:
> >
> > * i won't be switching things over until april 11th or 12th at the
> earliest.
> > *  this change will require a short (~15-20min) downtime to switch over
> from 3.4 to 3.6.
> > *  updating python will impact all active branches, so i will need to
> backport this pr:  https://github.com/apache/spark/pull/24266
> >
> > question:  which other branches should i be applying this change to?
> definitely master and 2.4...  also 2.3?  2.2?
> >
> > thanks in advance,
> >
> > shane
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>


-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

[SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

[shane knapp]

details here (see most recent comments for current state of things):
https://issues.apache.org/jira/browse/SPARK-25079

my PR for these changes:
https://github.com/apache/spark/pull/24266

we're doing this because we need to support arrow 0.12.1:
https://issues.apache.org/jira/browse/SPARK-27276

a couple of things:

* i won't be switching things over until april 11th or 12th at the earliest.
*  this change will require a short (~15-20min) downtime to switch over
from 3.4 to 3.6.
*  updating python will impact all active branches, so i will need to
backport this pr:  https://github.com/apache/spark/pull/24266

question:  which other branches should i be applying this change to?
definitely master and 2.4...  also 2.3?  2.2?

thanks in advance,

shane
-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: [SPARK-25079] moving from python 3.4 to python 3.6.8, impacts all active branches

[Sean Owen]

In theory Spark 2.4 supports Python 3.4; would this mean it's now just
tested vs 3.6? that's not out of the question, but can the older
branches continue to test on older versions or is that super complex?

On Wed, Apr 10, 2019 at 1:37 PM shane knapp  wrote:
>
> details here (see most recent comments for current state of things):
> https://issues.apache.org/jira/browse/SPARK-25079
>
> my PR for these changes:
> https://github.com/apache/spark/pull/24266
>
> we're doing this because we need to support arrow 0.12.1:
> https://issues.apache.org/jira/browse/SPARK-27276
>
> a couple of things:
>
> * i won't be switching things over until april 11th or 12th at the earliest.
> *  this change will require a short (~15-20min) downtime to switch over from 
> 3.4 to 3.6.
> *  updating python will impact all active branches, so i will need to 
> backport this pr:  https://github.com/apache/spark/pull/24266
>
> question:  which other branches should i be applying this change to?  
> definitely master and 2.4...  also 2.3?  2.2?
>
> thanks in advance,
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Support for Avro datasets

[Taoufik Dachraoui]

Hi

The PR #24299  proposes a
simple solution to add support for Avro objects; your opinions matter, we
are eager to have a working solution to create Avro datasets

regards
-- 
Taoufik Dachraoui

Joining of one stream against the most recent update of another stream

[Schneider, Thilo]

Hi,

I just opened a Jira task (https://issues.apache.org/jira/browse/SPARK-27424) 
outlining a specific streaming join and would like to raise your attention on 
this.

As this feature is a critical requirement for my current project, I am willing 
to work on that task. Before putting too much thought in it, however, I would 
like to ask you for your opinion / input / assessment. Do we need a formal SPIP 
for this?

Thanks
Thilo

Fraport AG Frankfurt Airport Services Worldwide, 60547 Frankfurt am Main, Sitz 
der Gesellschaft: Frankfurt am Main, Amtsgericht Frankfurt am Main: HRB 7042, 
Umsatzsteuer-Identifikationsnummer: DE 114150623, Vorsitzender des 
Aufsichtsrates: Karlheinz Weimar - Hessischer Finanzminister a.D.; Vorstand: 
Dr. Stefan Schulte (Vorsitzender), Anke Giesen, Michael Mueller, Dr. Matthias 
Zieschang