IntelliJ license for committers?

[Sean Owen]

I'm aware that IntelliJ has (at least in the past) made licenses
available to committers in bona fide open source projects, and I
recall they did the same for Spark. I believe I'm using that license
now, but it seems to have expired? If anyone knows the status of that
(or of any renewals to the license), I wonder if you could share that
with me, offline of course.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: IntelliJ license for committers?

[Yin Huai]

I think they can renew your license. In
https://www.jetbrains.com/buy/opensource/?product=idea, you can find
"Update Open Source License".

On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen  wrote:

> I'm aware that IntelliJ has (at least in the past) made licenses
> available to committers in bona fide open source projects, and I
> recall they did the same for Spark. I believe I'm using that license
> now, but it seems to have expired? If anyone knows the status of that
> (or of any renewals to the license), I wonder if you could share that
> with me, offline of course.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: IntelliJ license for committers?

[Sean Owen]

Thanks, yes I've seen this, though I recall from another project that
at some point they said, wait, we already gave your project a license!
and I had to track down who had it. I think Josh might be the keeper?
Not a big deal, just making sure I didn't miss an update there.

On Wed, Dec 2, 2015 at 6:18 PM, Yin Huai  wrote:
> I think they can renew your license. In
> https://www.jetbrains.com/buy/opensource/?product=idea, you can find "Update
> Open Source License".
>
> On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen  wrote:
>>
>> I'm aware that IntelliJ has (at least in the past) made licenses
>> available to committers in bona fide open source projects, and I
>> recall they did the same for Spark. I believe I'm using that license
>> now, but it seems to have expired? If anyone knows the status of that
>> (or of any renewals to the license), I wonder if you could share that
>> with me, offline of course.
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

[VOTE] Release Apache Spark 1.6.0 (RC1)

[Michael Armbrust]

Please vote on releasing the following candidate as Apache Spark version
1.6.0!

The vote is open until Saturday, December 5, 2015 at 21:00 UTC and passes
if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 1.6.0
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is *v1.6.0-rc1
(bf525845cef159d2d4c9f4d64e158f037179b5c4)
*

The release files, including signatures, digests, etc. can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-v1.6.0-rc1-bin/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1165/

The test repository (versioned as v1.6.0-rc1) for this release can be found
at:
https://repository.apache.org/content/repositories/orgapachespark-1164/

The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc1-docs/


===
== How can I help test this release? ==
===
If you are a Spark user, you can help us test this release by taking an
existing Spark workload and running on this release candidate, then
reporting any regressions.


== What justifies a -1 vote for this release? ==

This vote is happening towards the end of the 1.6 QA period, so -1 votes
should only occur for significant regressions from 1.5. Bugs already
present in 1.5, minor regressions, or bugs related to new features will not
block this release.

===
== What should happen to JIRA tickets still targeting 1.6.0? ==
===
1. It is OK for documentation patches to target 1.6.0 and still go into
branch-1.6, since documentations will be published separately from the
release.
2. New features for non-alpha-modules should target 1.7+.
3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the target
version.


==
== Major changes to help you focus your testing ==
==

Spark SQL

   - SPARK-10810 
   Session Management - The ability to create multiple isolated SQL
   Contexts that have their own configuration and default database.  This is
   turned on by default in the thrift server.
   - SPARK-   Dataset
   API - A type-safe API (similar to RDDs) that performs many operations on
   serialized binary data and code generation (i.e. Project Tungsten).
   - SPARK-1  Unified
   Memory Management - Shared memory for execution and caching instead of
   exclusive division of the regions.
   - SPARK-11197  SQL
   Queries on Files - Concise syntax for running SQL queries over files of
   any supported format without registering a table.
   - SPARK-11745  Reading
   non-standard JSON files - Added options to read non-standard JSON files
   (e.g. single-quotes, unquoted attributes)
   - SPARK-10412 
Per-operator
   Metics for SQL Execution - Display statistics on a per-operator basis
   for memory usage and spilled data size.
   - SPARK-11329  Star
   (*) expansion for StructTypes - Makes it easier to nest and unest
   arbitrary numbers of columns
   - SPARK-10917 ,
   SPARK-11149  In-memory
   Columnar Cache Performance - Significant (up to 14x) speed up when
   caching data that contains complex types in DataFrames or SQL.
   - SPARK-1  Fast
   null-safe joins - Joins using null-safe equality (<=>) will now execute
   using SortMergeJoin instead of computing a cartisian product.
   - SPARK-11389  SQL
   Execution Using Off-Heap Memory - Support for configuring query
   execution to occur using off-heap memory to avoid GC overhead
   - SPARK-10978  Datasource
   API Avoid Double Filter - When implementing a datasource with filter
   pushdown, developers can now tell Spark SQL to avoid double evaluating a
   pushed-down filter.
   - SPARK-4849   Advanced
   Layout of 

Re: IntelliJ license for committers?

[Josh Rosen]

Yep, I'm the point of contact between us and JetBrains. I forwarded the
2015 license renewal email to the private@ list, so it should be accessible
via the archives. I'll go ahead and forward you a copy of our project
license, which will have to be renewed in January of next year.

On Wed, Dec 2, 2015 at 10:24 AM, Sean Owen  wrote:

> Thanks, yes I've seen this, though I recall from another project that
> at some point they said, wait, we already gave your project a license!
> and I had to track down who had it. I think Josh might be the keeper?
> Not a big deal, just making sure I didn't miss an update there.
>
> On Wed, Dec 2, 2015 at 6:18 PM, Yin Huai  wrote:
> > I think they can renew your license. In
> > https://www.jetbrains.com/buy/opensource/?product=idea, you can find
> "Update
> > Open Source License".
> >
> > On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen  wrote:
> >>
> >> I'm aware that IntelliJ has (at least in the past) made licenses
> >> available to committers in bona fide open source projects, and I
> >> recall they did the same for Spark. I believe I'm using that license
> >> now, but it seems to have expired? If anyone knows the status of that
> >> (or of any renewals to the license), I wonder if you could share that
> >> with me, offline of course.
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >> For additional commands, e-mail: dev-h...@spark.apache.org
> >>
> >
>

When to cut RCs

[Sean Owen]

Moving this to dev --

That's good, that'll help. Technically there's still a Blocker bug:
https://issues.apache.org/jira/browse/SPARK-12000

The 'race' doesn't matter that much, but release planning remains the
real bug-bear here. There are still, for instance, 52 issues targeted
at 1.6.0, 42 of which were raised and targeted by committers. For
example, I count 6 'umbrella' JIRAs in ML alone that are still open.
The release is theoretically several weeks behind plan on what's
intended to be a fixed release cycle too. This is why I'm not sure why
today it's suddenly potentially ready for release.

I'm just curious, am I the only one that thinks this isn't roughly
normal, or do other people manage releases this way? I know the real
world is messy and this is better than in the past, but I still get
surprised by how each 1.x release actually comes about.

On Wed, Dec 2, 2015 at 8:12 PM, marmbrus  wrote:
> Github user marmbrus commented on the pull request:
>
> https://github.com/apache/spark/pull/10108#issuecomment-161420041
>
> Sorry if this seemed arbitrary, I cut the release as soon as all the 
> blocker issues were resolve.  In the future I'll announce on the dev list 
> first.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
> -
> To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
> For additional commands, e-mail: reviews-h...@spark.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [VOTE] Release Apache Spark 1.6.0 (RC1)

[Nicholas Chammas]

-0

If spark-ec2 is still a supported part of the project, then we should
update its version lists as new releases are made. 1.5.2 had the same issue.

https://github.com/apache/spark/blob/v1.6.0-rc1/ec2/spark_ec2.py#L54-L91

(I guess as part of the 2.0 discussions we should continue to discuss
whether spark-ec2 still belongs in the project. I'm starting to feel
awkward reporting spark-ec2 release issues...)

Nick

On Wed, Dec 2, 2015 at 3:27 PM Michael Armbrust 
wrote:

> Please vote on releasing the following candidate as Apache Spark version
> 1.6.0!
>
> The vote is open until Saturday, December 5, 2015 at 21:00 UTC and passes
> if a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.6.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see http://spark.apache.org/
>
> The tag to be voted on is *v1.6.0-rc1
> (bf525845cef159d2d4c9f4d64e158f037179b5c4)
> *
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-v1.6.0-rc1-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1165/
>
> The test repository (versioned as v1.6.0-rc1) for this release can be
> found at:
> https://repository.apache.org/content/repositories/orgapachespark-1164/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc1-docs/
>
>
> ===
> == How can I help test this release? ==
> ===
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> 
> == What justifies a -1 vote for this release? ==
> 
> This vote is happening towards the end of the 1.6 QA period, so -1 votes
> should only occur for significant regressions from 1.5. Bugs already
> present in 1.5, minor regressions, or bugs related to new features will not
> block this release.
>
> ===
> == What should happen to JIRA tickets still targeting 1.6.0? ==
> ===
> 1. It is OK for documentation patches to target 1.6.0 and still go into
> branch-1.6, since documentations will be published separately from the
> release.
> 2. New features for non-alpha-modules should target 1.7+.
> 3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the target
> version.
>
>
> ==
> == Major changes to help you focus your testing ==
> ==
>
> Spark SQL
>
>- SPARK-10810 
>Session Management - The ability to create multiple isolated SQL
>Contexts that have their own configuration and default database.  This is
>turned on by default in the thrift server.
>- SPARK-   Dataset
>API - A type-safe API (similar to RDDs) that performs many operations
>on serialized binary data and code generation (i.e. Project Tungsten).
>- SPARK-1  Unified
>Memory Management - Shared memory for execution and caching instead of
>exclusive division of the regions.
>- SPARK-11197  SQL
>Queries on Files - Concise syntax for running SQL queries over files
>of any supported format without registering a table.
>- SPARK-11745  Reading
>non-standard JSON files - Added options to read non-standard JSON
>files (e.g. single-quotes, unquoted attributes)
>- SPARK-10412  
> Per-operator
>Metics for SQL Execution - Display statistics on a per-operator basis
>for memory usage and spilled data size.
>- SPARK-11329  Star
>(*) expansion for StructTypes - Makes it easier to nest and unest
>arbitrary numbers of columns
>- SPARK-10917 ,
>SPARK-11149  In-memory
>Columnar Cache Performance - Significant (up to 14x) speed up when
>caching data that contains complex types in DataFrames or SQL.
>- SPARK-1  Fast
>null-safe joins 

Re: When to cut RCs

[Michael Armbrust]

>
> Sorry for a second email so soon. I meant to also ask, what keeps the cost
> of making an RC high? Can we bring it down with better tooling?
>

There is a lot of tooling:
https://amplab.cs.berkeley.edu/jenkins/view/Spark-Packaging/

Still you have check JIRA, sync with people who have been working on known
issues, run the jenkins jobs (which take 1+ hours) and then write that
email which has a bunch of links in it.  Short of automating the creation
of the email (PRs welcome!) I'm not sure what else you would automate.
That said, this is all I have done since I came into work today.

Re: When to cut RCs

[Sean Owen]

On Wed, Dec 2, 2015 at 9:06 PM, Michael Armbrust  wrote:
> This can be debated, but I explicitly ignored test and documentation issues.
> Since the docs are published separately and easy to update, I don't think
> its worth further disturbing the release cadence for these JIRAs.

It makes sense to not hold up an RC since they don't affect testing of
functionality. Prior releases have ultimately gone out with doc issues
still outstanding (and bugs) though. This doesn't seem to be on
anyone's release checklist, and maybe part of it is because they're
let slide for RCs.  Your suggestion to check-point release status
below sounds spot on; I sort of tried to do that earlier.


> Up until today various committers have told me that there were known issues
> with branch-1.6 that would cause them to -1 the release.  Whenever this
> happened, I asked them to ensure there was a properly targeted blocker JIRA
> open so people could publicly track the status of the release.  As long as
> such issues were open, I only published a preview since making an RC is
> pretty high cost.

Makes sense if these are all getting translated into Blockers and
resolved before an RC. It's the simplest mechanism to communicate and
track this in a distributed way.

"No blockers" is a minimal criterion for release. It still seems funny
to release with so many issues targeted for 1.6.0, including issues
that aren't critical or bugs. Sure, that's just hygiene. But without
it, do people take "Target Version" seriously? if they don't, is there
any force guiding people to prioritize or decide what to (not) work
on? I'm sure the communication happens, just doesn't seem like it's
fully on JIRA, which is ultimately suboptimal.


> I actually did spent quite a bit of time asking people to close various
> umbrella issues, and I was pretty strict about watching JIRA throughout the
> process.  Perhaps as an additional step, future preview releases or branch
> cuts can include a link to an authoritative dashboard that we will use to
> decide when we are ready to make an RC.  I'm also open to other suggestions.

Yes, that's great. It takes the same effort from everyone. Having a
green light on a dashboard at release time is only the symptom of
decent planning. The effect I think it really needs to have occurs
now: what's really probably on the menu for 1.7? and periodically
track against that goal. Then the release process is with any luck
just a formality with no surprises.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: When to cut RCs

[Sean Busbey]

On Wed, Dec 2, 2015 at 3:19 PM, Sean Busbey  wrote:

>
>
> On Wed, Dec 2, 2015 at 3:06 PM, Michael Armbrust 
> wrote:
>
>>
>>>
>> The release is theoretically several weeks behind plan on what's
>>> intended to be a fixed release cycle too. This is why I'm not sure why
>>> today it's suddenly potentially ready for release.
>>>
>>
>> Up until today various committers have told me that there were known
>> issues with branch-1.6 that would cause them to -1 the release.  Whenever
>> this happened, I asked them to ensure there was a properly targeted blocker
>> JIRA open so people could publicly track the status of the release.  As
>> long as such issues were open, I only published a preview since making an
>> RC is pretty high cost.
>>
>> I'm sorry that it felt sudden to you, but as of last night all such known
>> issues were resolved and thus I cut a release as soon as this was the case.
>>
>>
>
> It would help me, and I'm sure other less-active folks, if this kind of
> feedback cycle were visible on dev@spark. It would also have the benefit
> of getting feedback from non-committers who might e.g. have some issue that
> they see in their own production deployments.
>
>
>
Sorry for a second email so soon. I meant to also ask, what keeps the cost
of making an RC high? Can we bring it down with better tooling?


-- 
Sean

Re: When to cut RCs

[Sean Busbey]

On Wed, Dec 2, 2015 at 3:06 PM, Michael Armbrust 
wrote:

>
>>
> The release is theoretically several weeks behind plan on what's
>> intended to be a fixed release cycle too. This is why I'm not sure why
>> today it's suddenly potentially ready for release.
>>
>
> Up until today various committers have told me that there were known
> issues with branch-1.6 that would cause them to -1 the release.  Whenever
> this happened, I asked them to ensure there was a properly targeted blocker
> JIRA open so people could publicly track the status of the release.  As
> long as such issues were open, I only published a preview since making an
> RC is pretty high cost.
>
> I'm sorry that it felt sudden to you, but as of last night all such known
> issues were resolved and thus I cut a release as soon as this was the case.
>
>

It would help me, and I'm sure other less-active folks, if this kind of
feedback cycle were visible on dev@spark. It would also have the benefit of
getting feedback from non-committers who might e.g. have some issue that
they see in their own production deployments.



-- 
Sean

Re: IntelliJ license for committers?

[Sean Busbey]

the private@spark list is only available to PMC members[1]. Could we
document somewhere (the IntelliJ section of the wiki[2]?) both the current
point of contact and a list of what happens when things get renewed each
year?

That way we could include either a note that the POC should email the key
to all committers or that committers should reach out to the POC at a given
time?

[1]: PMC members and ASF Members, so Sean O specifically can search the
archives. but that won't solve the general problem.
[2]:
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ

On Wed, Dec 2, 2015 at 2:09 PM, Josh Rosen  wrote:

> Yep, I'm the point of contact between us and JetBrains. I forwarded the
> 2015 license renewal email to the private@ list, so it should be
> accessible via the archives. I'll go ahead and forward you a copy of our
> project license, which will have to be renewed in January of next year.
>
> On Wed, Dec 2, 2015 at 10:24 AM, Sean Owen  wrote:
>
>> Thanks, yes I've seen this, though I recall from another project that
>> at some point they said, wait, we already gave your project a license!
>> and I had to track down who had it. I think Josh might be the keeper?
>> Not a big deal, just making sure I didn't miss an update there.
>>
>> On Wed, Dec 2, 2015 at 6:18 PM, Yin Huai  wrote:
>> > I think they can renew your license. In
>> > https://www.jetbrains.com/buy/opensource/?product=idea, you can find
>> "Update
>> > Open Source License".
>> >
>> > On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen  wrote:
>> >>
>> >> I'm aware that IntelliJ has (at least in the past) made licenses
>> >> available to committers in bona fide open source projects, and I
>> >> recall they did the same for Spark. I believe I'm using that license
>> >> now, but it seems to have expired? If anyone knows the status of that
>> >> (or of any renewals to the license), I wonder if you could share that
>> >> with me, offline of course.
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> >> For additional commands, e-mail: dev-h...@spark.apache.org
>> >>
>> >
>>
>
>


-- 
Sean

Re: IntelliJ license for committers?

[Sean Owen]

Yeah I can see the PMC list as it happens; technically there are
committers that aren't PMC / ASF members though, yeah. Josh did update
the list when the last one expired, and the current license hasn't
expired yet, though it was no longer working for me. It turns out to
not be valid for the very newest IJ.

Josh actually gave me a better solution: just use the free IntelliJ.
Actually, none of the features we're likely to need are omitted from
the community edition.

Sure, it's cool if the existence of an update can be disseminated on
dev@ and I'm guessing it would certainly be when it came up. My
problem was actually slightly different and should have clarified I
was curious if anyone else had license problems 'early' and if I'd
missed an update, but I hadn't.

On Wed, Dec 2, 2015 at 9:26 PM, Sean Busbey  wrote:
> the private@spark list is only available to PMC members[1]. Could we
> document somewhere (the IntelliJ section of the wiki[2]?) both the current
> point of contact and a list of what happens when things get renewed each
> year?
>
> That way we could include either a note that the POC should email the key to
> all committers or that committers should reach out to the POC at a given
> time?
>
> [1]: PMC members and ASF Members, so Sean O specifically can search the
> archives. but that won't solve the general problem.
> [2]:
> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
>
> On Wed, Dec 2, 2015 at 2:09 PM, Josh Rosen  wrote:
>>
>> Yep, I'm the point of contact between us and JetBrains. I forwarded the
>> 2015 license renewal email to the private@ list, so it should be accessible
>> via the archives. I'll go ahead and forward you a copy of our project
>> license, which will have to be renewed in January of next year.
>>
>> On Wed, Dec 2, 2015 at 10:24 AM, Sean Owen  wrote:
>>>
>>> Thanks, yes I've seen this, though I recall from another project that
>>> at some point they said, wait, we already gave your project a license!
>>> and I had to track down who had it. I think Josh might be the keeper?
>>> Not a big deal, just making sure I didn't miss an update there.
>>>
>>> On Wed, Dec 2, 2015 at 6:18 PM, Yin Huai  wrote:
>>> > I think they can renew your license. In
>>> > https://www.jetbrains.com/buy/opensource/?product=idea, you can find
>>> > "Update
>>> > Open Source License".
>>> >
>>> > On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen  wrote:
>>> >>
>>> >> I'm aware that IntelliJ has (at least in the past) made licenses
>>> >> available to committers in bona fide open source projects, and I
>>> >> recall they did the same for Spark. I believe I'm using that license
>>> >> now, but it seems to have expired? If anyone knows the status of that
>>> >> (or of any renewals to the license), I wonder if you could share that
>>> >> with me, offline of course.
>>> >>
>>> >> -
>>> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>>> >> For additional commands, e-mail: dev-h...@spark.apache.org
>>> >>
>>> >
>>
>>
>
>
>
> --
> Sean

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: IntelliJ license for committers?

[Luciano Resende]

On Wed, Dec 2, 2015 at 3:02 PM, Reynold Xin  wrote:

> For IntelliJ I think the free version is sufficient for Spark development.
>
>
And I believe anyone with a @apache.org e-mail address can request their
own personal license for InteliJ. That's what  I personally did.

-- 
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [VOTE] Release Apache Spark 1.6.0 (RC1)

[Michael Armbrust]

I'm going to kick the voting off with a +1 (binding).  We ran TPC-DS and
most queries are faster than 1.5.  We've also ported several production
pipelines to 1.6.

[build system] jenkins downtime, thursday 12/10/15 7am PDT

[shane knapp]

there's Yet Another Jenkins Security Advisory[tm], and a big release
to patch it all coming out next wednesday.

to that end i will be performing a jenkins update, as well as
performing the work to resolve the following jira issue:
https://issues.apache.org/jira/browse/SPARK-11255

i will put jenkins in to quiet mode around 6am, start work around 7am
and expect everything to be back up and building before 9am.  i'll
post updates as things progress.

please let me know ASAP if there's any problem with this schedule.

shane

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: When to cut RCs

[Patrick Wendell]

In terms of advertising to people the status of the release and whether an
RC is likely to go out, the best mechanism I can think of is our current
mechanism of using JIRA and respecting the semantics of a blocker JIRA. We
could do a better job though creating a JIRA dashboard for each release and
linking to it publicly so it's very clear to people what is going on. I
have always used one privately when managing previous releases, but no
reason we can't put one up on the website or wiki.

IMO a mailing list is not a great mechanism for the fine-grained work of
release management because of the sheer complexity and volume of finalizing
a spark release. Being a release manager means tracking over a course of
several weeks typically dozens of distinct issues and trying to prioritize
them, get more clarity from the report of those issues, possibly reaching
out to people on the phone or in person to get more details, etc. You want
a mutable dashboard where you can convey the current status clearly.

What might be good in the early stages is a weekly e-mail to the dev@ list
just refreshing what is on the JIRA and letting people know how things are
looking. So someone just passing by has some idea of how things are going
and can chime in, etc.

Once an RC is cut then we do mostly rely on the mailing list for
discussion. At that point the number of known issues is small enough I
think to discuss in an all-to-all fashion.

- Patrick

On Wed, Dec 2, 2015 at 1:25 PM, Sean Owen  wrote:

> On Wed, Dec 2, 2015 at 9:06 PM, Michael Armbrust 
> wrote:
> > This can be debated, but I explicitly ignored test and documentation
> issues.
> > Since the docs are published separately and easy to update, I don't think
> > its worth further disturbing the release cadence for these JIRAs.
>
> It makes sense to not hold up an RC since they don't affect testing of
> functionality. Prior releases have ultimately gone out with doc issues
> still outstanding (and bugs) though. This doesn't seem to be on
> anyone's release checklist, and maybe part of it is because they're
> let slide for RCs.  Your suggestion to check-point release status
> below sounds spot on; I sort of tried to do that earlier.
>
>
> > Up until today various committers have told me that there were known
> issues
> > with branch-1.6 that would cause them to -1 the release.  Whenever this
> > happened, I asked them to ensure there was a properly targeted blocker
> JIRA
> > open so people could publicly track the status of the release.  As long
> as
> > such issues were open, I only published a preview since making an RC is
> > pretty high cost.
>
> Makes sense if these are all getting translated into Blockers and
> resolved before an RC. It's the simplest mechanism to communicate and
> track this in a distributed way.
>
> "No blockers" is a minimal criterion for release. It still seems funny
> to release with so many issues targeted for 1.6.0, including issues
> that aren't critical or bugs. Sure, that's just hygiene. But without
> it, do people take "Target Version" seriously? if they don't, is there
> any force guiding people to prioritize or decide what to (not) work
> on? I'm sure the communication happens, just doesn't seem like it's
> fully on JIRA, which is ultimately suboptimal.
>
>
> > I actually did spent quite a bit of time asking people to close various
> > umbrella issues, and I was pretty strict about watching JIRA throughout
> the
> > process.  Perhaps as an additional step, future preview releases or
> branch
> > cuts can include a link to an authoritative dashboard that we will use to
> > decide when we are ready to make an RC.  I'm also open to other
> suggestions.
>
> Yes, that's great. It takes the same effort from everyone. Having a
> green light on a dashboard at release time is only the symptom of
> decent planning. The effect I think it really needs to have occurs
> now: what's really probably on the menu for 1.7? and periodically
> track against that goal. Then the release process is with any luck
> just a formality with no surprises.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: Python API for Association Rules

[Caique Marques]

Hi Joseph.
Sorry for my fail, I will comment on Jira.

Thanks.
Caique.

2015-12-02 19:12 GMT-02:00 Joseph Bradley :

> If you're working on a feature, please comment on the JIRA first (to avoid
> conflicts / duplicate work).  Could you please copy what your wrote to the
> JIRA to discuss there?
> Thanks,
> Joseph
>
> On Wed, Dec 2, 2015 at 4:51 AM, caiquermarques95 <
> caiquermarque...@gmail.com> wrote:
>
>> Hello everyone!
>> I'm developing to the Python API for association rules (
>> https://issues.apache.org/jira/browse/SPARK-8855), but I found a doubt.
>>
>> Following the description of the issue, it says that a important method
>> is "*FPGrowthModel.generateAssociationRules()*", of course. However, is
>> not clear if a wrapper for the association rules it will be in "
>> *FPGrowthModelWrapper.scala*" and this is the problem.
>>
>> My idea is the following:
>> 1) In the fpm.py file; class "Association Rules" with one method and a
>> class:
>> 1.1) Method train(data, minConfidence), that will generate the
>> association rules for a data with a minConfidence specified (0.6 default).
>> This method will call the "trainAssociationRules" from the
>> *PythonMLLibAPI* with the parameters data and minConfidence. Later. will
>> return a FPGrowthModel.
>> 1.2) Class Rule, that will a namedtuple, represents an (antecedent,
>> consequent) tuple.
>>
>> 2) Still in fpm.py, in the class FPGrowthModel, a new method will be
>> added, called generateAssociationRules, that will map the Rules generated
>> calling the method "getAssociationRule" from FPGrowthModelWrapper to the
>> namedtuple.
>>
>> Now is my doubt, how to make trainAssociationRules returns a FGrowthModel
>> to the Wrapper just maps the rule received to the antecedent/consequent? I
>> could not do the method trainAssociationRules returns a FPGrowthModel. The
>> wrapper for association rules is in FPGrowthModelWrapper, right?
>>
>> For illustration, I think something like this in *PythonMLLibAPI:*
>>
>> def trainAssociationRules(
>>   data: JavaRDD[FPGrowth.FreqItemset[Any]],
>>   minConfidence: Double): [return type] = {
>>
>> val model = new FPGrowthModel(data.rdd)
>>   .generateAssociationRules(minConfidence)
>>
>> new FPGrowthModelWrapper(model)
>>   }
>>
>> And in FPGrowthModelWrapper, something like:
>>
>>  def getAssociationRules: [return type] = {
>> SerDe.fromTuple2RDD(rule.map(x => (x.javaAntecedent,
>> x.javaConsequent)))
>>  }
>>
>> I know that will fail, but, what is wrong with my idea?
>> Any suggestions?
>>
>> Thanks for the help and the tips.
>> Caique.
>>
>> --
>> View this message in context: Python API for Association Rules
>> 
>> Sent from the Apache Spark Developers List mailing list archive
>>  at
>> Nabble.com.
>>
>
>

Re: Multiplication on decimals in a dataframe query

[Akhil Das]

Not quiet sure whats happening, but its not an issue with multiplication i
guess as the following query worked for me:

trades.select(trades("price")*9.5).show
+-+
|(price * 9.5)|
+-+
|199.5|
|228.0|
|190.0|
|199.5|
|190.0|
|256.5|
|218.5|
|275.5|
|218.5|
..
..


Could it be with the precision? ccing dev list, may be you can open up a
jira for this as it seems to be a bug.

Thanks
Best Regards

On Mon, Nov 30, 2015 at 12:41 AM, Philip Dodds 
wrote:

> I hit a weird issue when I tried to multiply to decimals in a select
> (either in scala or as SQL), and Im assuming I must be missing the point.
>
> The issue is fairly easy to recreate with something like the following:
>
>
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
> import sqlContext.implicits._
> import org.apache.spark.sql.types.Decimal
>
> case class Trade(quantity: Decimal,price: Decimal)
>
> val data = Seq.fill(100) {
>   val price = Decimal(20+scala.util.Random.nextInt(10))
> val quantity = Decimal(20+scala.util.Random.nextInt(10))
>
>   Trade(quantity, price)
> }
>
> val trades = sc.parallelize(data).toDF()
> trades.registerTempTable("trades")
>
> trades.select(trades("price")*trades("quantity")).show
>
> sqlContext.sql("select
> price/quantity,price*quantity,price+quantity,price-quantity from
> trades").show
>
> The odd part is if you run it you will see that the addition/division and
> subtraction works but the multiplication returns a null.
>
> Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)
>
> ie.
>
> +--+
>
> |(price * quantity)|
>
> +--+
>
> |  null|
>
> |  null|
>
> |  null|
>
> |  null|
>
> |  null|
>
> +--+
>
>
> +++++
>
> | _c0| _c1| _c2| _c3|
>
> +++++
>
> |0.952380952380952381|null|41.00...|-1.00...|
>
> |1.380952380952380952|null|50.00...|8.00|
>
> |1.272727272727272727|null|50.00...|6.00|
>
> |0.83|null|44.00...|-4.00...|
>
> |1.00|null|58.00...|   0E-18|
>
> +++++
>
>
> Just keen to know what I did wrong?
>
>
> Cheers
>
> P
>
> --
> Philip Dodds
>
>
>

Re: IntelliJ license for committers?

[Reynold Xin]

For IntelliJ I think the free version is sufficient for Spark development.

On Thursday, December 3, 2015, Sean Owen  wrote:

> Yeah I can see the PMC list as it happens; technically there are
> committers that aren't PMC / ASF members though, yeah. Josh did update
> the list when the last one expired, and the current license hasn't
> expired yet, though it was no longer working for me. It turns out to
> not be valid for the very newest IJ.
>
> Josh actually gave me a better solution: just use the free IntelliJ.
> Actually, none of the features we're likely to need are omitted from
> the community edition.
>
> Sure, it's cool if the existence of an update can be disseminated on
> dev@ and I'm guessing it would certainly be when it came up. My
> problem was actually slightly different and should have clarified I
> was curious if anyone else had license problems 'early' and if I'd
> missed an update, but I hadn't.
>
> On Wed, Dec 2, 2015 at 9:26 PM, Sean Busbey  > wrote:
> > the private@spark list is only available to PMC members[1]. Could we
> > document somewhere (the IntelliJ section of the wiki[2]?) both the
> current
> > point of contact and a list of what happens when things get renewed each
> > year?
> >
> > That way we could include either a note that the POC should email the
> key to
> > all committers or that committers should reach out to the POC at a given
> > time?
> >
> > [1]: PMC members and ASF Members, so Sean O specifically can search the
> > archives. but that won't solve the general problem.
> > [2]:
> >
> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
> >
> > On Wed, Dec 2, 2015 at 2:09 PM, Josh Rosen  > wrote:
> >>
> >> Yep, I'm the point of contact between us and JetBrains. I forwarded the
> >> 2015 license renewal email to the private@ list, so it should be
> accessible
> >> via the archives. I'll go ahead and forward you a copy of our project
> >> license, which will have to be renewed in January of next year.
> >>
> >> On Wed, Dec 2, 2015 at 10:24 AM, Sean Owen  > wrote:
> >>>
> >>> Thanks, yes I've seen this, though I recall from another project that
> >>> at some point they said, wait, we already gave your project a license!
> >>> and I had to track down who had it. I think Josh might be the keeper?
> >>> Not a big deal, just making sure I didn't miss an update there.
> >>>
> >>> On Wed, Dec 2, 2015 at 6:18 PM, Yin Huai  > wrote:
> >>> > I think they can renew your license. In
> >>> > https://www.jetbrains.com/buy/opensource/?product=idea, you can find
> >>> > "Update
> >>> > Open Source License".
> >>> >
> >>> > On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen  > wrote:
> >>> >>
> >>> >> I'm aware that IntelliJ has (at least in the past) made licenses
> >>> >> available to committers in bona fide open source projects, and I
> >>> >> recall they did the same for Spark. I believe I'm using that license
> >>> >> now, but it seems to have expired? If anyone knows the status of
> that
> >>> >> (or of any renewals to the license), I wonder if you could share
> that
> >>> >> with me, offline of course.
> >>> >>
> >>> >>
> -
> >>> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> 
> >>> >> For additional commands, e-mail: dev-h...@spark.apache.org
> 
> >>> >>
> >>> >
> >>
> >>
> >
> >
> >
> > --
> > Sean
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> For additional commands, e-mail: dev-h...@spark.apache.org 
>
>

Re: [VOTE] Release Apache Spark 1.6.0 (RC1)

[Ted Yu]

I tried to run test suite and encountered the following:

http://pastebin.com/DPnwMGrm

FYI

On Wed, Dec 2, 2015 at 12:39 PM, Nicholas Chammas <
nicholas.cham...@gmail.com> wrote:

> -0
>
> If spark-ec2 is still a supported part of the project, then we should
> update its version lists as new releases are made. 1.5.2 had the same issue.
>
> https://github.com/apache/spark/blob/v1.6.0-rc1/ec2/spark_ec2.py#L54-L91
>
> (I guess as part of the 2.0 discussions we should continue to discuss
> whether spark-ec2 still belongs in the project. I'm starting to feel
> awkward reporting spark-ec2 release issues...)
>
> Nick
>
> On Wed, Dec 2, 2015 at 3:27 PM Michael Armbrust 
> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 1.6.0!
>>
>> The vote is open until Saturday, December 5, 2015 at 21:00 UTC and passes
>> if a majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 1.6.0
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see http://spark.apache.org/
>>
>> The tag to be voted on is *v1.6.0-rc1
>> (bf525845cef159d2d4c9f4d64e158f037179b5c4)
>> *
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-v1.6.0-rc1-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1165/
>>
>> The test repository (versioned as v1.6.0-rc1) for this release can be
>> found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1164/
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc1-docs/
>>
>>
>> ===
>> == How can I help test this release? ==
>> ===
>> If you are a Spark user, you can help us test this release by taking an
>> existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> 
>> == What justifies a -1 vote for this release? ==
>> 
>> This vote is happening towards the end of the 1.6 QA period, so -1 votes
>> should only occur for significant regressions from 1.5. Bugs already
>> present in 1.5, minor regressions, or bugs related to new features will not
>> block this release.
>>
>> ===
>> == What should happen to JIRA tickets still targeting 1.6.0? ==
>> ===
>> 1. It is OK for documentation patches to target 1.6.0 and still go into
>> branch-1.6, since documentations will be published separately from the
>> release.
>> 2. New features for non-alpha-modules should target 1.7+.
>> 3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the target
>> version.
>>
>>
>> ==
>> == Major changes to help you focus your testing ==
>> ==
>>
>> Spark SQL
>>
>>- SPARK-10810 
>>Session Management - The ability to create multiple isolated SQL
>>Contexts that have their own configuration and default database.  This is
>>turned on by default in the thrift server.
>>- SPARK-   Dataset
>>API - A type-safe API (similar to RDDs) that performs many operations
>>on serialized binary data and code generation (i.e. Project Tungsten).
>>- SPARK-1  Unified
>>Memory Management - Shared memory for execution and caching instead
>>of exclusive division of the regions.
>>- SPARK-11197  SQL
>>Queries on Files - Concise syntax for running SQL queries over files
>>of any supported format without registering a table.
>>- SPARK-11745  Reading
>>non-standard JSON files - Added options to read non-standard JSON
>>files (e.g. single-quotes, unquoted attributes)
>>- SPARK-10412  
>> Per-operator
>>Metics for SQL Execution - Display statistics on a per-operator basis
>>for memory usage and spilled data size.
>>- SPARK-11329  Star
>>(*) expansion for StructTypes - Makes it easier to nest and unest
>>arbitrary numbers of columns
>>- SPARK-10917 ,
>>SPARK-11149 

Python API for Association Rules

[caiquermarques95]

Hello everyone!
I'm developing to the Python API for association rules (
https://issues.apache.org/jira/browse/SPARK-8855), but I found a doubt.

Following the description of the issue, it says that a important method is "
*FPGrowthModel.generateAssociationRules()*", of course. However, is not
clear if a wrapper for the association rules it will be in "
*FPGrowthModelWrapper.scala*" and this is the problem.

My idea is the following:
1) In the fpm.py file; class "Association Rules" with one method and a
class:
1.1) Method train(data, minConfidence), that will generate the association
rules for a data with a minConfidence specified (0.6 default). This method
will call the "trainAssociationRules" from the *PythonMLLibAPI* with the
parameters data and minConfidence. Later. will return a FPGrowthModel.
1.2) Class Rule, that will a namedtuple, represents an (antecedent,
consequent) tuple.

2) Still in fpm.py, in the class FPGrowthModel, a new method will be added,
called generateAssociationRules, that will map the Rules generated calling
the method "getAssociationRule" from FPGrowthModelWrapper to the namedtuple.

Now is my doubt, how to make trainAssociationRules returns a FGrowthModel
to the Wrapper just maps the rule received to the antecedent/consequent? I
could not do the method trainAssociationRules returns a FPGrowthModel. The
wrapper for association rules is in FPGrowthModelWrapper, right?

For illustration, I think something like this in *PythonMLLibAPI:*

def trainAssociationRules(
  data: JavaRDD[FPGrowth.FreqItemset[Any]],
  minConfidence: Double): [return type] = {

val model = new FPGrowthModel(data.rdd)
  .generateAssociationRules(minConfidence)

new FPGrowthModelWrapper(model)
  }

And in FPGrowthModelWrapper, something like:

 def getAssociationRules: [return type] = {
SerDe.fromTuple2RDD(rule.map(x => (x.javaAntecedent, x.javaConsequent)))
 }

I know that will fail, but, what is wrong with my idea?
Any suggestions?

Thanks for the help and the tips.
Caique.




--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Python-API-for-Association-Rules-tp15419.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: When to cut RCs

[Michael Armbrust]

Thanks for bringing this up Sean. I think we are all happy to adopt
concrete suggestions to make the release process more transparent,
including pinging the list before kicking off the release build.

Technically there's still a Blocker bug:
> https://issues.apache.org/jira/browse/SPARK-12000


Sorry, I misprioritized this particular issue when I thought that it was
going to block the release by causing the doc build to fail. When I
realized the failure was non-deterministic and isolated to OSX (i.e. the
official release build on jenkins is not affected) I failed to update the
issue.  It doesn't show up on the dashboard that I've been using to track
the release

since
its labeled documentation.


> The 'race' doesn't matter that much, but release planning remains the
> real bug-bear here. There are still, for instance, 52 issues targeted
> at 1.6.0, 42 of which were raised and targeted by committers. For
> example, I count 6 'umbrella' JIRAs in ML alone that are still open.
>

This can be debated, but I explicitly ignored test and documentation
issues.  Since the docs are published separately and easy to update, I
don't think its worth further disturbing the release cadence for these
JIRAs.


> The release is theoretically several weeks behind plan on what's
> intended to be a fixed release cycle too. This is why I'm not sure why
> today it's suddenly potentially ready for release.
>

Up until today various committers have told me that there were known issues
with branch-1.6 that would cause them to -1 the release.  Whenever this
happened, I asked them to ensure there was a properly targeted blocker JIRA
open so people could publicly track the status of the release.  As long as
such issues were open, I only published a preview since making an RC is
pretty high cost.

I'm sorry that it felt sudden to you, but as of last night all such known
issues were resolved and thus I cut a release as soon as this was the case.

I'm just curious, am I the only one that thinks this isn't roughly
> normal, or do other people manage releases this way? I know the real
> world is messy and this is better than in the past, but I still get
> surprised by how each 1.x release actually comes about.
>

I actually did spent quite a bit of time asking people to close various
umbrella issues, and I was pretty strict about watching JIRA throughout the
process.  Perhaps as an additional step, future preview releases or branch
cuts can include a link to an authoritative dashboard that we will use to
decide when we are ready to make an RC.  I'm also open to other suggestions.

Michael

Re: Python API for Association Rules

[Joseph Bradley]

If you're working on a feature, please comment on the JIRA first (to avoid
conflicts / duplicate work).  Could you please copy what your wrote to the
JIRA to discuss there?
Thanks,
Joseph

On Wed, Dec 2, 2015 at 4:51 AM, caiquermarques95  wrote:

> Hello everyone!
> I'm developing to the Python API for association rules (
> https://issues.apache.org/jira/browse/SPARK-8855), but I found a doubt.
>
> Following the description of the issue, it says that a important method is
> "*FPGrowthModel.generateAssociationRules()*", of course. However, is not
> clear if a wrapper for the association rules it will be in "
> *FPGrowthModelWrapper.scala*" and this is the problem.
>
> My idea is the following:
> 1) In the fpm.py file; class "Association Rules" with one method and a
> class:
> 1.1) Method train(data, minConfidence), that will generate the association
> rules for a data with a minConfidence specified (0.6 default). This method
> will call the "trainAssociationRules" from the *PythonMLLibAPI* with the
> parameters data and minConfidence. Later. will return a FPGrowthModel.
> 1.2) Class Rule, that will a namedtuple, represents an (antecedent,
> consequent) tuple.
>
> 2) Still in fpm.py, in the class FPGrowthModel, a new method will be
> added, called generateAssociationRules, that will map the Rules generated
> calling the method "getAssociationRule" from FPGrowthModelWrapper to the
> namedtuple.
>
> Now is my doubt, how to make trainAssociationRules returns a FGrowthModel
> to the Wrapper just maps the rule received to the antecedent/consequent? I
> could not do the method trainAssociationRules returns a FPGrowthModel. The
> wrapper for association rules is in FPGrowthModelWrapper, right?
>
> For illustration, I think something like this in *PythonMLLibAPI:*
>
> def trainAssociationRules(
>   data: JavaRDD[FPGrowth.FreqItemset[Any]],
>   minConfidence: Double): [return type] = {
>
> val model = new FPGrowthModel(data.rdd)
>   .generateAssociationRules(minConfidence)
>
> new FPGrowthModelWrapper(model)
>   }
>
> And in FPGrowthModelWrapper, something like:
>
>  def getAssociationRules: [return type] = {
> SerDe.fromTuple2RDD(rule.map(x => (x.javaAntecedent,
> x.javaConsequent)))
>  }
>
> I know that will fail, but, what is wrong with my idea?
> Any suggestions?
>
> Thanks for the help and the tips.
> Caique.
>
> --
> View this message in context: Python API for Association Rules
> 
> Sent from the Apache Spark Developers List mailing list archive
>  at
> Nabble.com.
>

Re: query on SVD++

[Yanbo Liang]

You means the SVDPlusPlus in GraphX? If you want to use SVD++ to train CF
model, I recommend you to use ALS which is more efficiency and has python
interface.

2015-12-02 11:21 GMT+08:00 张志强(旺轩) :

> Hi All,
>
>
>
> I came across the SVD++ algorithm implementation in Spark code base, but I
> was wondering why we didn’t expose the scala api interface to python?
>
> Any plan to do this?
>
>
>
> BR,
>
> -Allen Zhang
>

Re: [VOTE] Release Apache Spark 1.6.0 (RC1)

[Ted Yu]

+1

Ran through test suite (minus docker-integration-tests) which passed.

Overall experience was much better compared with some of the prior RC's.

[INFO] Spark Project External Kafka ... SUCCESS [
53.956 s]
[INFO] Spark Project Examples . SUCCESS [02:05
min]
[INFO] Spark Project External Kafka Assembly .. SUCCESS [
11.298 s]
[INFO]

[INFO] BUILD SUCCESS
[INFO]

[INFO] Total time: 01:42 h
[INFO] Finished at: 2015-12-02T17:19:02-08:00

On Wed, Dec 2, 2015 at 4:23 PM, Michael Armbrust 
wrote:

> I'm going to kick the voting off with a +1 (binding).  We ran TPC-DS and
> most queries are faster than 1.5.  We've also ported several production
> pipelines to 1.6.
>