[
https://issues.apache.org/jira/browse/SPARK-60206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126533#comment-18126533
]
Lisa Cao commented on SPARK-60206:
----------------------------------
[~linhongliu-db] https://github.com/apache/spark/pull/59416 fixes the source:
tables, SQL sources and filter subqueries now resolve in the creation
namespace, and temp views aren't picked up.
Functions in dimension and measure expressions still resolve in the querying
session, because those expressions are spliced into the query plan. For
example, a dimension {{f(k)}} calls {{other.f}} after {{USE other}}. Resolving
them inside the metric view first didn't work for SQL functions, and qualifying
function names would duplicate function resolution. How would you like to
handle it? Happy to do the follow-up.
> Metric views resolve unqualified names in the reader's current namespace
> instead of the creation namespace
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60206
> URL: https://issues.apache.org/jira/browse/SPARK-60206
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Lisa Cao
> Priority: Major
> Labels: pull-request-available
>
> A metric view resolves unqualified names against the current catalog and
> namespace of whoever queries it, not the ones it was created in. Regular
> views use the creation namespace.
> {code:sql}
> CREATE TABLE default.t USING parquet AS SELECT * FROM VALUES ('a', 1), ('b',
> 2) AS v(k, x);
> CREATE DATABASE other;
> CREATE TABLE other.t USING parquet AS SELECT * FROM VALUES ('z', 1000) AS
> v(k, x);
> USE default;
> CREATE VIEW default.mv WITH METRICS LANGUAGE YAML AS $$
> version: "0.1"
> source: t
> dimensions:
> - name: k
> expr: k
> measures:
> - name: total
> expr: sum(x)
> $$;
> CREATE VIEW default.v AS SELECT sum(x) AS total FROM t;
> USE other;
> SELECT measure(total) FROM default.mv; -- 1000, reads other.t
> SELECT total FROM default.v; -- 3
> {code}
> The same happens for:
> * SQL sources ({{source: SELECT * FROM t}})
> * functions in dimension expressions ({{expr: f(k)}} calls {{other.f}})
> * subqueries in {{filter}}
> * temp views: with {{CREATE TEMPORARY VIEW t}} (or
> {{df.createOrReplaceTempView("t")}}) in the reader's session, the metric view
> reads the temp view, even from {{default}}
> * v2 catalog metric views, including after {{USE}} of another catalog
> From a namespace without {{t}} it fails with {{TABLE_OR_VIEW_NOT_FOUND}}.
> Reproduced on 4.2.0 and 4.3.0-rc1, with in-memory and Hive catalogs, and over
> Spark Connect.
> The creation namespace is stored ({{view.catalogAndNamespace.*}}), but
> {{SessionCatalog.parseMetricViewDefinition}} only applies the view's SQL
> configs. Regular views resolve under
> {{AnalysisContext.withAnalysisContext(view.desc)}} ({{ViewResolution}});
> metric views skip it.
> Workaround: fully qualify names in the definition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
