GitHub user nongli opened a pull request:
https://github.com/apache/spark/pull/12007
[SPARK-14210][SQL] Add a metric for time spent in scans.
## What changes were proposed in this pull request?
This adds a metric to parquet scans that measures the time in just the scan
phase. This is
only possible when the scan returns ColumnarBatches, otherwise the overhead
is too high.
This combined with the pipeline metric lets us easily see what percent of
the time was
in the scan.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nongli/spark spark-14210
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12007.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12007
----
commit b17bac4dd880bc93455d85f9591468941a88ce97
Author: Nong Li <[email protected]>
Date: 2016-03-25T22:47:19Z
[SPARK-14210][SQL] Add a metric for time spent in scans.
This adds a metric to parquet scans that measures the time in just the scan
phase. This is
only possible when the scan returns ColumnarBatches, otherwise the overhead
is too high.
This combined with the pipeline metric lets us easily see what percent of
the time was
in the scan.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
