GitHub user ericl opened a pull request:
https://github.com/apache/spark/pull/13456
[SPARK-15724] Add benchmarks for performance over wide schemas
## What changes were proposed in this pull request?
This adds microbenchmarks for tracking performance of queries over very
wide or deeply nested DataFrames. It seems performance seriously degrades when
DataFrames get thousands of columns wide or hundreds of fields deep.
## How was this patch tested?
Current results included.
cc @rxin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ericl/spark sc-3468
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13456.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13456
----
commit a2dfaa2defa247ae1a8bce8fb3d4ddb769c5ea81
Author: Eric Liang <[email protected]>
Date: 2016-06-02T00:36:06Z
Wed Jun 1 17:36:06 PDT 2016
commit 6feb80134993cbafdc38aae52e64826a70ba3fb6
Author: Eric Liang <[email protected]>
Date: 2016-06-02T00:39:08Z
Wed Jun 1 17:39:08 PDT 2016
commit b6fe715cf5ce66490611bfa5bce9770f1338493f
Author: Eric Liang <[email protected]>
Date: 2016-06-02T00:45:03Z
Wed Jun 1 17:45:03 PDT 2016
commit c6e96526dd08b3bd43b8d4c58f9d669cdff34b93
Author: Eric Liang <[email protected]>
Date: 2016-06-02T00:45:13Z
Wed Jun 1 17:45:13 PDT 2016
commit af543b9a5d3ebe8aa369358d1eefaa02d3a9de19
Author: Eric Liang <[email protected]>
Date: 2016-06-02T00:45:17Z
Wed Jun 1 17:45:17 PDT 2016
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
