GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/17998
[SPARK-20703][SQL][WIP] Add an operator for writing data out
## What changes were proposed in this pull request?
Right now in the explain plan / UI, we have no way to tell whether a query
is writing data out, and also there is no way to associate metrics with data
writes. We should add an operator for writing data out. This operator can be
used to track writing data out and related metrics.
### Approach
We have several paths for writing data out through some `RunnableCommand`
classes.
#### File-based relation: `InsertIntoHadoopFsRelationCommand`,
`InsertIntoHiveTable`
Those commands use `FileFormatWriter` to write out data files. We can
record some metrics in `FileFormatWriter` and update it later.
`FileFormatWriter` accepts a `QueryExecution`. We can track the execution plan
of `QueryExecution`.
This patch adds a new operator `WriteDataFileOutExec`. It is simply used to
track the metrics of writing data file out for file-based relations. Currently
we track some metrics:
* number of written files
* number of dynamic partitions
* bytes of written files
* number of output rows
* writing data out time (ms)
#### Other datasources: `InsertIntoDataSourceCommand`,
`SaveIntoDataSourceCommand`
For other datasource relations, the logic of writing data out is delegated
to the datasource implementations, e.g., `InsertableRelation.insert`,
`CreatableRelationProvider.createRelation`. The APIs basically take a
`DataFrame` for the data to write. Those APIs can possibly create new
`DataFrame` based on the give one. So we can't easily track its execution, and
theoretically we don't know the details of the API implementation in those
datasources. So we can't obtain enough metrics.
## How was this patch tested?
Jenkins tests.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 SPARK-20703
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17998.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17998
----
commit f60e2801ba79eceefcc5c1dc2a9e1a3fbd026534
Author: Liang-Chi Hsieh <[email protected]>
Date: 2017-05-14T11:11:40Z
Add an operator for writing data out.
commit eb8553e105f197c46e3e99745b8aa17dac8f1105
Author: Liang-Chi Hsieh <[email protected]>
Date: 2017-05-15T08:56:22Z
Set metrics for file-based relation.
commit afb6e752922b6b2270b1e598f86f20ca2a858e51
Author: Liang-Chi Hsieh <[email protected]>
Date: 2017-05-16T09:57:13Z
Add tests for writing data metrics.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
