JingsongLi opened a new pull request, #11132:
URL: https://github.com/apache/paimon/pull/11132

   ### Purpose
   
   Add a read-only Spark 4.2 execution profile for Ossie `0.3.0.dev0` semantic 
views stored in Paimon REST Catalog, on top of #11127.
   
   ```sql
   SELECT region, MEASURE(revenue)
   FROM paimon.sales.order_metrics
   GROUP BY region;
   ```
   
   The resolver reads definitions through the existing semantic-view management 
client, validates the pinned Ossie schema, binds dataset fields separately from 
source columns, and compiles the model into a native Spark Metric View. Spark 
performs the `MEASURE` rewrite and query execution. Authentication, REST prefix 
and source authorization use the existing catalog paths. Ordinary tables 
introduced while expanding SQL views are left for Spark's normal relation 
resolution.
   
   Definitions are read for each new analysis; already-analyzed queries retain 
their plan. Querying does not persist a converted model or rewrite the original 
YAML. The schema is pinned to Ossie commit 
`2692b1cc4708fdcf6a91e597ca928a9bc9d0da60`, which matches the current upstream 
schema despite the mutable development version.
   
   This initial profile supports one physical Paimon dataset, scalar fields, 
dataset/model metrics and an explicit portable expression subset. 
Relationships, multiple datasets, joins against a semantic relation, streaming, 
writes and semantic-view SQL DDL are outside the profile. Unsupported execution 
features are rejected. The REST server still needs to implement semantic-view 
storage and authorization.
   
   The schema validator and its date/time dependency are shaded into the 4.2 
connector. Safe YAML loading preserves scalar types long enough to reject 
non-JSON extension values, non-string keys and recursive aliases.
   
   ### Tests
   
   - `mvn -pl paimon-spark/paimon-spark-4.2 -am -Pspark42 -DfailIfNoTests=false 
-Dtest=none 
-DwildcardSuites=org.apache.paimon.spark.ossie.OssieModelTest,org.apache.spark.sql.paimon.OssieSemanticViewTest,org.apache.paimon.spark.sql.Spark42CompatibilityTest
 package`: 29 passed with normal format/checkstyle/enforcer checks enabled.
   - Repeated all 29 tests with the shaded connector JAR first on the runtime 
classpath.
   - 14 Ossie tests cover schema/version validation, JSON-compatible YAML data, 
field/source shadowing, dataset and model metric binding, DISTINCT/FILTER, 
null/empty results, division/rounding, alias collisions, namespace binding, 
definition replacement, metadata preservation, permission/HTTP errors and 
unsupported operations.
   - The existing 15 Spark 4.2 compatibility tests cover normal 
table/view/function and V1/V2 row-level paths. They also caught and guard 
against semantic lookup intercepting source tables introduced by ordinary view 
expansion.
   - Root Apache RAT and `git diff --check` passed.
   
   The complete multi-version Spark test suite has not been run for this 
4.2-specific change.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to