Uros Stankovic created SPARK-47226:
--------------------------------------
Summary: Move DataSourceMetricsMixin from JDBC RDD to JDBC Relation
Key: SPARK-47226
URL: https://issues.apache.org/jira/browse/SPARK-47226
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Uros Stankovic
We added DataSourceMetricsMixin trait recently which is holder for sequence of
SQLMetric objects.
We added it to JDBCRDD, so rdd can define which metrics it has and then those
metrics would be added to metrics of RowDataSourceScanExec.
The problem that can appear in the future is that someone will want to do some
transformation on RDD, like mapPartitions which returns new object of RDD that
does not implement our interface. Since we cannot access parent of RDD, it is
impossible to get those metrics in RowDataSourceScanExec.
The better solution would be to make relations implement mentioned trait and
that should solve our problem, same trait can be used, because it is not linked
with RDD by name or by any property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
