Re: [VOTE] Spark 2.1.2 (RC4)

2017-10-04 Thread Felix Cheung 
+1

Tested SparkR package manually on multiple platforms and checked different 
Hadoop release jar.

And previously tested the last RC on different R releases (see the last RC vote 
thread)

I found some differences in bin release jars created by the different options 
when running the make-release script, created this JIRA to track
https://issues.apache.org/jira/browse/SPARK-22202

I've checked to confirm these exist in 2.1.1 release so this isn't a 
regression, and hence my +1.

btw, I think we need to update this file for the new keys used in signing this 
release https://www.apache.org/dist/spark/KEYS


_
From: Liwei Lin >
Sent: Wednesday, October 4, 2017 6:51 PM
Subject: Re: [VOTE] Spark 2.1.2 (RC4)
To: Spark dev list >


+1 (non-binding)


Cheers,
Liwei

On Wed, Oct 4, 2017 at 4:03 PM, Nick Pentreath 
> wrote:
Ah right! Was using a new cloud instance and didn't realize I was logged in as 
root! thanks

On Tue, 3 Oct 2017 at 21:13 Marcelo Vanzin 
> wrote:
Maybe you're running as root (or the admin account on your OS)?

On Tue, Oct 3, 2017 at 12:12 PM, Nick Pentreath
> wrote:
> Hmm I'm consistently getting this error in core tests:
>
> - SPARK-3697: ignore directories that cannot be read. *** FAILED ***
>   2 was not equal to 1 (FsHistoryProviderSuite.scala:146)
>
>
> Anyone else? Any insight? Perhaps it's my set up.
>
>>>
>>>
>>> On Tue, Oct 3, 2017 at 7:24 AM Holden Karau 
>>> > wrote:

 Please vote on releasing the following candidate as Apache Spark version
 2.1.2. The vote is open until Saturday October 7th at 9:00 PST and passes 
 if
 a majority of at least 3 +1 PMC votes are cast.

 [ ] +1 Release this package as Apache Spark 2.1.2
 [ ] -1 Do not release this package because ...


 To learn more about Apache Spark, please see https://spark.apache.org/

 The tag to be voted on is v2.1.2-rc4
 (2abaea9e40fce81cd4626498e0f5c28a70917499)

 List of JIRA tickets resolved in this release can be found with this
 filter.

 The release files, including signatures, digests, etc. can be found at:
 https://home.apache.org/~holden/spark-2.1.2-rc4-bin/

 Release artifacts are signed with a key from:
 https://people.apache.org/~holden/holdens_keys.asc

 The staging repository for this release can be found at:
 https://repository.apache.org/content/repositories/orgapachespark-1252

 The documentation corresponding to this release can be found at:
 https://people.apache.org/~holden/spark-2.1.2-rc4-docs/


 FAQ

 How can I help test this release?

 If you are a Spark user, you can help us test this release by taking an
 existing Spark workload and running on this release candidate, then
 reporting any regressions.

 If you're working in PySpark you can set up a virtual env and install
 the current RC and see if anything important breaks, in the Java/Scala you
 can add the staging repository to your projects resolvers and test with the
 RC (make sure to clean up the artifact cache before/after so you don't end
 up building with a out of date RC going forward).

 What should happen to JIRA tickets still targeting 2.1.2?

 Committers should look at those and triage. Extremely important bug
 fixes, documentation, and API tweaks that impact compatibility should be
 worked on immediately. Everything else please retarget to 2.1.3.

 But my bug isn't fixed!??!

 In order to make timely releases, we will typically not hold the release
 unless the bug in question is a regression from 2.1.1. That being said if
 there is something which is a regression form 2.1.1 that has not been
 correctly targeted please ping a committer to help target the issue (you 
 can
 see the open issues listed as impacting Spark 2.1.1 & 2.1.2)

 What are the unresolved issues targeted for 2.1.2?

 At this time there are no open unresolved issues.

 Is there anything different about this release?

 This is the first release in awhile not built on the AMPLAB Jenkins.
 This is good because it means future releases can more easily be built and
 signed securely (and I've been updating the documentation in
 https://github.com/apache/spark-website/pull/66 as I progress), however the
 chances of a mistake are higher with any change like this. If there
 something you normally take for granted as correct when checking a release,
 please double check this time :)

 Should I be committing code to branch-2.1?

 Thanks for asking!

Re: Nightly builds for master branch failed

2017-10-04 Thread Felix Cheung 
Hmm, sounds like some sort of corruption of the maven directory on the Jenkins 
box...



From: Liwei Lin 
Sent: Wednesday, October 4, 2017 6:52:54 PM
To: Spark dev list
Subject: Nightly builds for master branch failed

https://amplab.cs.berkeley.edu/jenkins/job/spark-master-maven-snapshots/

Nightly builds for master branch failed due to:

[error] error: error reading 
/home/jenkins/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.7.1/jackson-databind-2.6.7.1.jar;
 zip file is empty

Can we get it fixed please? Thanks!



Cheers,
Liwei

Re: Disabling Closed -> Reopened transition for non-committers

2017-10-04 Thread Felix Cheung 
To be sure, this is only for JIRA and not for github PR, right?

If then +1 but I think the access control on JIRA does not necessarily match 
the committer list, and is manually maintained, last I hear.


From: Sean Owen 
Sent: Wednesday, October 4, 2017 7:51:37 PM
To: Dongjoon Hyun
Cc: dev
Subject: Re: Disabling Closed -> Reopened transition for non-committers

Although I assume we could get an account suspended if it started opening spam 
issues, yes we default to letting anyone open issues, and potentially abusing 
it. That much is the right default and I don't see any policy tweak that stops 
that.

I see several INFRA tickets asking to *allow* the Closed -> Reopened 
transition, which suggests it's not the default. 
https://issues.apache.org/jira/browse/INFRA-11857?jql=project%20%3D%20INFRA%20AND%20text%20~%20%22reopen%20JIRA%22

I'm accustomed to Closed being a final state that nobody can reopen as a matter 
of workflow -- the idea being that anything else should be a new discussion if 
the current issue was deemed formally done.

Spark pretty much leaves all issues in "Resolved" status which can still be 
reopened, and I think that's right. Although I'd like to limit all reopening to 
committers, it isn't that important.

Being able to move a JIRA to Closed permanently seems useful, as it doesn't 
interfere with any normal workflow, doesn't actually prevent a new issue from 
succeeding it in normal usage, and gives another tool to limit a specific kind 
of abuse.

On Thu, Oct 5, 2017 at 3:28 AM Dongjoon Hyun 
> wrote:
It can stop reopening, but new JIRA issues with duplicate content will be 
created intentionally instead.

Is that policy (privileged reopening) used in other Apache communities for that 
purpose?


On Wed, Oct 4, 2017 at 7:06 PM, Sean Owen 
> wrote:
We have this problem occasionally, where a disgruntled user continually reopens 
an issue after it's closed.

https://issues.apache.org/jira/browse/SPARK-21999

(Feel free to comment on this one if anyone disagrees)

Regardless of that particular JIRA, I'd like to disable to Closed -> Reopened 
transition for non-committers: https://issues.apache.org/jira/browse/INFRA-15221

Re: Disabling Closed -> Reopened transition for non-committers

2017-10-04 Thread Sean Owen 
Although I assume we could get an account suspended if it started opening
spam issues, yes we default to letting anyone open issues, and potentially
abusing it. That much is the right default and I don't see any policy tweak
that stops that.

I see several INFRA tickets asking to *allow* the Closed -> Reopened
transition, which suggests it's not the default.
https://issues.apache.org/jira/browse/INFRA-11857?jql=project%20%3D%20INFRA%20AND%20text%20~%20%22reopen%20JIRA%22

I'm accustomed to Closed being a final state that nobody can reopen as a
matter of workflow -- the idea being that anything else should be a new
discussion if the current issue was deemed formally done.

Spark pretty much leaves all issues in "Resolved" status which can still be
reopened, and I think that's right. Although I'd like to limit all
reopening to committers, it isn't that important.

Being able to move a JIRA to Closed permanently seems useful, as it doesn't
interfere with any normal workflow, doesn't actually prevent a new issue
from succeeding it in normal usage, and gives another tool to limit a
specific kind of abuse.

On Thu, Oct 5, 2017 at 3:28 AM Dongjoon Hyun 
wrote:

> It can stop reopening, but new JIRA issues with duplicate content will be
> created intentionally instead.
>
> Is that policy (privileged reopening) used in other Apache communities for
> that purpose?
>
>
> On Wed, Oct 4, 2017 at 7:06 PM, Sean Owen  wrote:
>
>> We have this problem occasionally, where a disgruntled user continually
>> reopens an issue after it's closed.
>>
>> https://issues.apache.org/jira/browse/SPARK-21999
>>
>> (Feel free to comment on this one if anyone disagrees)
>>
>> Regardless of that particular JIRA, I'd like to disable to Closed ->
>> Reopened transition for non-committers:
>> https://issues.apache.org/jira/browse/INFRA-15221
>>
>>
>

Re: Disabling Closed -> Reopened transition for non-committers

2017-10-04 Thread Dongjoon Hyun 
It can stop reopening, but new JIRA issues with duplicate content will be
created intentionally instead.

Is that policy (privileged reopening) used in other Apache communities for
that purpose?


On Wed, Oct 4, 2017 at 7:06 PM, Sean Owen  wrote:

> We have this problem occasionally, where a disgruntled user continually
> reopens an issue after it's closed.
>
> https://issues.apache.org/jira/browse/SPARK-21999
>
> (Feel free to comment on this one if anyone disagrees)
>
> Regardless of that particular JIRA, I'd like to disable to Closed ->
> Reopened transition for non-committers: https://issues.apache.org/
> jira/browse/INFRA-15221
>
>

Disabling Closed -> Reopened transition for non-committers

2017-10-04 Thread Sean Owen 
We have this problem occasionally, where a disgruntled user continually
reopens an issue after it's closed.

https://issues.apache.org/jira/browse/SPARK-21999

(Feel free to comment on this one if anyone disagrees)

Regardless of that particular JIRA, I'd like to disable to Closed ->
Reopened transition for non-committers:
https://issues.apache.org/jira/browse/INFRA-15221

Re: Welcoming Tejas Patil as a Spark committer

2017-10-04 Thread Liwei Lin 
Congratulations!

Cheers,
Liwei

On Wed, Oct 4, 2017 at 2:27 PM, Yuval Itzchakov  wrote:

> Congratulations and Good luck! :)
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Nightly builds for master branch failed

2017-10-04 Thread Liwei Lin 
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-maven-snapshots/

Nightly builds for master branch failed due to:

[error] error: error reading
/home/jenkins/.m2/repository/com/fasterxml/jackson/core/jackson-databind/
2.6.7.1/jackson-databind-2.6.7.1.jar; zip file is empty

Can we get it fixed please? Thanks!



Cheers,
Liwei

Re: [VOTE] Spark 2.1.2 (RC4)

2017-10-04 Thread Liwei Lin 
+1 (non-binding)


Cheers,
Liwei

On Wed, Oct 4, 2017 at 4:03 PM, Nick Pentreath 
wrote:

> Ah right! Was using a new cloud instance and didn't realize I was logged
> in as root! thanks
>
> On Tue, 3 Oct 2017 at 21:13 Marcelo Vanzin  wrote:
>
>> Maybe you're running as root (or the admin account on your OS)?
>>
>> On Tue, Oct 3, 2017 at 12:12 PM, Nick Pentreath
>>  wrote:
>> > Hmm I'm consistently getting this error in core tests:
>> >
>> > - SPARK-3697: ignore directories that cannot be read. *** FAILED ***
>> >   2 was not equal to 1 (FsHistoryProviderSuite.scala:146)
>> >
>> >
>> > Anyone else? Any insight? Perhaps it's my set up.
>> >
>> >>>
>> >>>
>> >>> On Tue, Oct 3, 2017 at 7:24 AM Holden Karau 
>> wrote:
>> 
>>  Please vote on releasing the following candidate as Apache Spark
>> version
>>  2.1.2. The vote is open until Saturday October 7th at 9:00 PST and
>> passes if
>>  a majority of at least 3 +1 PMC votes are cast.
>> 
>>  [ ] +1 Release this package as Apache Spark 2.1.2
>>  [ ] -1 Do not release this package because ...
>> 
>> 
>>  To learn more about Apache Spark, please see
>> https://spark.apache.org/
>> 
>>  The tag to be voted on is v2.1.2-rc4
>>  (2abaea9e40fce81cd4626498e0f5c28a70917499)
>> 
>>  List of JIRA tickets resolved in this release can be found with this
>>  filter.
>> 
>>  The release files, including signatures, digests, etc. can be found
>> at:
>>  https://home.apache.org/~holden/spark-2.1.2-rc4-bin/
>> 
>>  Release artifacts are signed with a key from:
>>  https://people.apache.org/~holden/holdens_keys.asc
>> 
>>  The staging repository for this release can be found at:
>>  https://repository.apache.org/content/repositories/
>> orgapachespark-1252
>> 
>>  The documentation corresponding to this release can be found at:
>>  https://people.apache.org/~holden/spark-2.1.2-rc4-docs/
>> 
>> 
>>  FAQ
>> 
>>  How can I help test this release?
>> 
>>  If you are a Spark user, you can help us test this release by taking
>> an
>>  existing Spark workload and running on this release candidate, then
>>  reporting any regressions.
>> 
>>  If you're working in PySpark you can set up a virtual env and install
>>  the current RC and see if anything important breaks, in the
>> Java/Scala you
>>  can add the staging repository to your projects resolvers and test
>> with the
>>  RC (make sure to clean up the artifact cache before/after so you
>> don't end
>>  up building with a out of date RC going forward).
>> 
>>  What should happen to JIRA tickets still targeting 2.1.2?
>> 
>>  Committers should look at those and triage. Extremely important bug
>>  fixes, documentation, and API tweaks that impact compatibility
>> should be
>>  worked on immediately. Everything else please retarget to 2.1.3.
>> 
>>  But my bug isn't fixed!??!
>> 
>>  In order to make timely releases, we will typically not hold the
>> release
>>  unless the bug in question is a regression from 2.1.1. That being
>> said if
>>  there is something which is a regression form 2.1.1 that has not been
>>  correctly targeted please ping a committer to help target the issue
>> (you can
>>  see the open issues listed as impacting Spark 2.1.1 & 2.1.2)
>> 
>>  What are the unresolved issues targeted for 2.1.2?
>> 
>>  At this time there are no open unresolved issues.
>> 
>>  Is there anything different about this release?
>> 
>>  This is the first release in awhile not built on the AMPLAB Jenkins.
>>  This is good because it means future releases can more easily be
>> built and
>>  signed securely (and I've been updating the documentation in
>>  https://github.com/apache/spark-website/pull/66 as I progress),
>> however the
>>  chances of a mistake are higher with any change like this. If there
>>  something you normally take for granted as correct when checking a
>> release,
>>  please double check this time :)
>> 
>>  Should I be committing code to branch-2.1?
>> 
>>  Thanks for asking! Please treat this stage in the RC process as "code
>>  freeze" so bug fixes only. If you're uncertain if something should
>> be back
>>  ported please reach out. If you do commit to branch-2.1 please tag
>> your JIRA
>>  issue fix version for 2.1.3 and if we cut another RC I'll move the
>> 2.1.3
>>  fixed into 2.1.2 as appropriate.
>> 
>>  What happened to RC3?
>> 
>>  Some R+zinc interactions kept it from getting out the door.
>>  --
>>  Twitter: https://twitter.com/holdenkarau
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>

Re: Interested to Contribute in Spark Development

2017-10-04 Thread Reynold Xin 
Kumar,

This is a good start: http://spark.apache.org/contributing.html


On Wed, Oct 4, 2017 at 10:00 AM, vaquar khan  wrote:

> Hi Nishant,
>
> 1) Start with helping spark users on mailing list and stack .
>
> 2) Start helping build and testing.
>
> 3) Once comfortable with code start working on Spark Jira.
>
>
> Regards,
> Vaquar khan
>
> On Oct 4, 2017 11:29 AM, "Kumar Nishant"  wrote:
>
>> Hi Team,
>> I am new to Apache community and I would love to contribute effort in
>> Spark development. Can anyone mentor & guide me how to proceed and start
>> contributing? I am beginner here so I am not sure what process is be
>> followed.
>>
>> Thanks
>> Nishant
>>
>>

Re: Interested to Contribute in Spark Development

2017-10-04 Thread vaquar khan 
Hi Nishant,

1) Start with helping spark users on mailing list and stack .

2) Start helping build and testing.

3) Once comfortable with code start working on Spark Jira.


Regards,
Vaquar khan

On Oct 4, 2017 11:29 AM, "Kumar Nishant"  wrote:

> Hi Team,
> I am new to Apache community and I would love to contribute effort in
> Spark development. Can anyone mentor & guide me how to proceed and start
> contributing? I am beginner here so I am not sure what process is be
> followed.
>
> Thanks
> Nishant
>
>

Interested to Contribute in Spark Development

2017-10-04 Thread Kumar Nishant 
Hi Team,
I am new to Apache community and I would love to contribute effort in Spark
development. Can anyone mentor & guide me how to proceed and start
contributing? I am beginner here so I am not sure what process is be
followed.

Thanks
Nishant

Re: [VOTE] Spark 2.1.2 (RC4)

2017-10-04 Thread Nick Pentreath 
Ah right! Was using a new cloud instance and didn't realize I was logged in
as root! thanks

On Tue, 3 Oct 2017 at 21:13 Marcelo Vanzin  wrote:

> Maybe you're running as root (or the admin account on your OS)?
>
> On Tue, Oct 3, 2017 at 12:12 PM, Nick Pentreath
>  wrote:
> > Hmm I'm consistently getting this error in core tests:
> >
> > - SPARK-3697: ignore directories that cannot be read. *** FAILED ***
> >   2 was not equal to 1 (FsHistoryProviderSuite.scala:146)
> >
> >
> > Anyone else? Any insight? Perhaps it's my set up.
> >
> >>>
> >>>
> >>> On Tue, Oct 3, 2017 at 7:24 AM Holden Karau 
> wrote:
> 
>  Please vote on releasing the following candidate as Apache Spark
> version
>  2.1.2. The vote is open until Saturday October 7th at 9:00 PST and
> passes if
>  a majority of at least 3 +1 PMC votes are cast.
> 
>  [ ] +1 Release this package as Apache Spark 2.1.2
>  [ ] -1 Do not release this package because ...
> 
> 
>  To learn more about Apache Spark, please see
> https://spark.apache.org/
> 
>  The tag to be voted on is v2.1.2-rc4
>  (2abaea9e40fce81cd4626498e0f5c28a70917499)
> 
>  List of JIRA tickets resolved in this release can be found with this
>  filter.
> 
>  The release files, including signatures, digests, etc. can be found
> at:
>  https://home.apache.org/~holden/spark-2.1.2-rc4-bin/
> 
>  Release artifacts are signed with a key from:
>  https://people.apache.org/~holden/holdens_keys.asc
> 
>  The staging repository for this release can be found at:
> 
> https://repository.apache.org/content/repositories/orgapachespark-1252
> 
>  The documentation corresponding to this release can be found at:
>  https://people.apache.org/~holden/spark-2.1.2-rc4-docs/
> 
> 
>  FAQ
> 
>  How can I help test this release?
> 
>  If you are a Spark user, you can help us test this release by taking
> an
>  existing Spark workload and running on this release candidate, then
>  reporting any regressions.
> 
>  If you're working in PySpark you can set up a virtual env and install
>  the current RC and see if anything important breaks, in the
> Java/Scala you
>  can add the staging repository to your projects resolvers and test
> with the
>  RC (make sure to clean up the artifact cache before/after so you
> don't end
>  up building with a out of date RC going forward).
> 
>  What should happen to JIRA tickets still targeting 2.1.2?
> 
>  Committers should look at those and triage. Extremely important bug
>  fixes, documentation, and API tweaks that impact compatibility should
> be
>  worked on immediately. Everything else please retarget to 2.1.3.
> 
>  But my bug isn't fixed!??!
> 
>  In order to make timely releases, we will typically not hold the
> release
>  unless the bug in question is a regression from 2.1.1. That being
> said if
>  there is something which is a regression form 2.1.1 that has not been
>  correctly targeted please ping a committer to help target the issue
> (you can
>  see the open issues listed as impacting Spark 2.1.1 & 2.1.2)
> 
>  What are the unresolved issues targeted for 2.1.2?
> 
>  At this time there are no open unresolved issues.
> 
>  Is there anything different about this release?
> 
>  This is the first release in awhile not built on the AMPLAB Jenkins.
>  This is good because it means future releases can more easily be
> built and
>  signed securely (and I've been updating the documentation in
>  https://github.com/apache/spark-website/pull/66 as I progress),
> however the
>  chances of a mistake are higher with any change like this. If there
>  something you normally take for granted as correct when checking a
> release,
>  please double check this time :)
> 
>  Should I be committing code to branch-2.1?
> 
>  Thanks for asking! Please treat this stage in the RC process as "code
>  freeze" so bug fixes only. If you're uncertain if something should be
> back
>  ported please reach out. If you do commit to branch-2.1 please tag
> your JIRA
>  issue fix version for 2.1.3 and if we cut another RC I'll move the
> 2.1.3
>  fixed into 2.1.2 as appropriate.
> 
>  What happened to RC3?
> 
>  Some R+zinc interactions kept it from getting out the door.
>  --
>  Twitter: https://twitter.com/holdenkarau
> >>
> >>
> >
>
>
>
> --
> Marcelo
>

Re: Welcoming Tejas Patil as a Spark committer

2017-10-04 Thread Yuval Itzchakov 
Congratulations and Good luck! :)



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org